You are currently browsing the tag archive for the 'assessment' tag.
Meriem Chida arrived in the CTLT offices a week after Fall’08 semester had begun, a new hire with an idea to bring her problem-based teaching methods to Pullman. Chida had connections with experts in her industry, Fashion Forecasting, and had used a design project with expert feedback in her previous teaching.
Pullman was remote from her experts, so she was forced to pilot her methods online, and have students create “posters” that could be shared with one another and experts in portfolios (blogs). While the technologies have changed, the outlines of these ideas go back a decade.
With CTLT, Chida piloted the Harvesting Gradebook , a technique to gather structures and open-ended feedback from her three audiences: industry, faculty in the department and student peers. In addition to gathering feedback about the student projects, Chida’s “gradebook” gathered feedback on the assessment tools (rubrics) she used. This allowed the community to answer — is this way of assessing how students talk about data useful? And her work helped explore the different character of feedback provided by industry, faculty and student peers. Chida is receiving one of the Faculty Teaching, Learning, and Assessment Innovation Awards for her Scholarship of Teaching and Learning. She is exploring the development of community-based learning, where the community has a role in the discussion about what is important to learn and how it might be best assessed.
We have been writing about an idea for a new approach to assessment of student work taking place in Web 2.0 environments. Gary Brown coined the term Harvesting Gradebook to describe this idea.
You are invited to try the Harvesting Gradebook as a member of the world community. This time-sensitive opportunity is part of a Washington State University course happening during Spring 2009.
Please understand that you are working with real students and an instructor in an actual running course. We appreciate your thoughtful and sensitive feedback. This blog also welcomes your meta-comments on the idea and the experience.
PROCESS
The instructor’s blog contains the assignment a series of posts which serve as the assignment prompt beginning here and adding some style guides and then some clarifications about the blog content and the assessment criteria and then a topic list. The balance of the instructor’s blog are personal reflections and modeling of activities in the assignment.
The instructor’s blog roll provides a link to the student blogs. In them you will see multiple posts by students, inviting traditional blog comments, and (in many cases) with an invitation to a “survey” that will look something like this:
Please evaluate this post:
http://skylight.wsu.edu/s/xxxxx.srv
You can preview the survey (opens in new window) (don’t post review data here, this is a just a preview ). Previewing is worth while because you will be asked to reflect on the assignment prompt, the student work and the value of the rubric itself. This 360-degree review is a reason we are calling this a transformative assessment approach.
Understand this is a work in progress, we are revising based on feedback and our own reflection. Also, understand that we are attempting to model the process by which this kind of process is evolved by the community using it, so your feedback to us (comment or trackback) is valuable. Please give us meta-comments on the process that are not appropriate for the feedback form itself.
If this experience whets your appetite to learn more about our thinking, a collection of materials can be found in a round table we prepared for AAC&U in January .
In July we made a proposal for a presentation for the AAC&U conference in January 2009 (which has been accepted, see you in Seattle). A previous post serves as an index to the materials we are collecting for that presentation. The AAC&U call for proposals asked a series of questions that reveal their perspective on the roles of the institution, faculty, students and the wider community in the learning enterprise.
The assumptions and perspectives that appear to underlie AAC&U’s framing of the questions are summarized in the table below. The preamble to the call for proposals also expresses a concern for maintaining the relevance of colleges and universities and maintaining the importance of the values of a liberal education. We see a conflict between the AAC&U assumptions and the goal of maintaining relevance in a Web 2.0 world.
The work we are reporting at the conference comes from a different set of assumptions about the learning enterprise. The perspective is community centered. It uses a set of assumptions that have previously existed within institutions of higher learning, and to some extent, still exist at the PhD level, but which seem to have been displaced by the challenges of increasing scale and decreasing resources. We are exploring ways in which the Internet, and particularly the Web 2.0 perspective of the Internet, can facilitate a transformation to a different set of assumptions.
Central to this thinking is our work from last summer on a Harvesting (or Transformative) Grade book. An important aspect of this re-visioning of the grade book is its move into a Web 2.0 context and its role in facilitating our assumptions about learning in communities (right column in table). This work is related to ideas Stephen Downes shared about group vs network learning and the lists below are worth examining in light of his diagram of that idea. This little video about connectivism gives some other ideas about faculty roles and implementation of our community-centric assumptions. Some very good comments about the role of the faculty appears ¾ of the way into the video.
| Institution-centric (AAC&U) view | Community-centric View |
| The institution and faculty are the central node. | The learner is the central node; learners include all members of the discourse community (or Community of Practice). |
| The faculty member is the grading authority. | Community of Practice holds responsibility (and the social capital) for assessment, and assessment (from the root “assay” as in to sit with) is most useful when it is formative, when it is understood to be constructive feedback rather than an authority’s judgment. |
| Faculty define the assessment criteria. | Community of Practice, by virtue of expert consensus, validates the assessment instrument. |
| Feedback to the student about learning criteria is masked by the letter grade. | Students merit feedback direct and unfiltered from the community using criteria that the community articulates. |
| Institution is the established (credentialing) authority. | Community of Practice is the implicit credentialing authority; the university is the facilitator of that credentialing and of community building. |
| The unit of analysis is the course, bounded in time, content, and brick or virtual (LMS) space. | The unit of analysis is the problem, problems are not bound in time or content. |
| Problems for students to study are (artificially) constrained by course and disciplinary boundaries. | Communities identify authentic problems that are interdisciplinary and reach beyond the definition of the course. |
| Students rarely share their work with, or receive feedback from, public audiences. | ePortfolios built over multiple year are learner owned (not university owned) and used to communicate (collect, select, connect, project, and reflect) with and get feedback from wider communities. |
| Faculty are a gatekeeper between the students and the community of practice. | Students are anticipated to join communities of practice, faculty may introduce students to the community. |
| Faculty have all the social capital within both the classroom and the community of practice. | (Student) learning is social and therefore learners are building social capital within the classroom and withing the community of practice. |
Related posts
1. Pilot course using the Harvesting Gradebook
2. Rich assessment data from a Harvesting Gradebook
3. Community conversation about assessment criteria
4. Differences in engaging the discussion among participant groups
5. Different conversations among participant groups
6. Evidence for Impact on Learning
First experience
This is one in a series of posts of material being prepared for presentation at the AAC&U conference in January 2009
The examples in the following posts are all drawn from a course offered at Washington State University in Fall 2008. We have previously demonstrated implementations of the gradebook, but none were practical for course use. In the course described below, we used our online survey tool, Skylight, to create an implementation that (while still labor intensive) was feasible for a pilot. We have ideas for a more robust implementation of the gradebook process that would make the process scalable to enterprise levels.
The course involved 87 students in 14 teams working on a semester-long design project. Each team created an electronic poster presentation (see poster template) of their work at midterm for formative feedback and then created a revised electronic poster as a final project. The instructor recruited industry professionals and other instructors to give feedback to the posters using a rubric derived from WSU’s critical and integrative thinking rubric. Students used the rubric for self-evaluation and also to give peer feedback.
Students were able to see results of the feedback in real time, via a dashboard we adapted from a tool WSU uses to returning course evaluation results to instructors.
Student teams were each working in team blogs (Blogger) where they compiled research information and other materials required for the course project. The instructor and other faculty, and the industry experts had access to the blogs. The project posters were placed in a blog post, along with a URL to the feedback survey (this was the data entry mechanism for the Harvesting Gradebook).
The instructor, industry experts, and peers from other teams gave feedback using the feedback survey at mid-term. Student teams were able to use this information to revise their work, which was then reviewed by 7 faculty from the program, 6 industry experts, and peers.
The following reports will demonstrate that the Harvesting Gradebook can serve as an assessment tool capable of distinguishing among student performance (as hypothesized last summer) and that it is a tool that can facilitate a conversation among stakeholders about the criteria used in the assessment, in addition to a conversation about the work being assessed. The former we contend are important elements for a community’s discussion of the learning of its participants.
This is one in a series of posts of material being prepared for presentation for the AAC&U conference in January 2009.
In July we postulated a new type of grade book that would have more potential for learners and learning than the traditional grade book does because it can open up richer forms of assessment. Included in that description was an illustration of the grade book, showing that it was conceivable student performances could be rendered in ways that would be both descriptive of students learning strengths and deficits and also informative to instructors and programs about the efficacy of the assignments.
In our pilot course this semester, we were able to gather data that confirms one of our hypotheses from the summer – student work can be rendered in descriptive ways and with enough resolution to tell students apart by their various abilities. The tables below compare the self- and industry ratings, and self- and faculty ratings on the seven dimensions of the rubric used. And also shows the hiring decision (a holistic judgment) by the self-, faculty- and industry- raters.
Figure 1. Student Team 1 assessment data from all seven dimensions of the rubric, comparing self-assessment to industry and to faculty assessment. In each of the figures above, the rubric dimensions are as follows: 1. Ability to identify the problem; 2. Ability to consider context; 3. OWN perspective, hypothesis or position is developed and communicated; 4. Ability to select, evaluate and interpret data; 5. Integration of OTHER (disciplinary) perspectives and positions; 6. Identifies and assesses conclusions, implications, and consequences; 7. Communicates effectively. For more information on the radar plot see this.
Figure 2. Student Team 2 assessment data from all seven dimensions of the rubric, comparing self-assessment to industry and to faculty assessment. Rubric dimensions are the same as figure 1.
Reporting the data to students
The system provided each student team real-time access to their assessments using a “dashboard” that students could log into to retrieve reports from the rubric surveys. The dashboard was originally designed to return online course evaluation results to faculty. This tool is analogous in function to the online gradebook in a learning management system, in that it shows a student one row of grades and the instructor a row for each student.
Analysis of the data
No activities to norm with the rubric were included with the class procedures. The lack of norming is evident in the data. While lack of norming of raters is a limitation of this pilot, within groups (students, faculty, and industry) raters are fairly well normed. The lack of intra-group norming might be seen as an occasion for conversation. Faculty and industry are in closer agreement as groups, but faculty tended to be a bit more generous.
The figures above also show the hiring decision (an aggregate assessment) made by each rater. Industry and faculty ratings on the rubric (below competency) correspond to their employability assessments, internship or less.
Comparing midterm to final there is evidence of growth in the eyes of industry and faculty, though the growth is in different directions and at different rates for each student team.
It is worth noting, students are over-confident of their abilities, consistent with their novice role within the community of practice. Most students also reflected their self-perception of learning growth during the course as increasing their competence. In a few cases, there was evidence that as students learned during the course they reduced their self-assessments, moving more in line with the assessments of industry and faculty.
This is one in a series of posts of material being prepared for presentation for the AAC&U conference in January 2009.
In July we postulated a new type of grade book that would have more potential for learners and the community of practice to converse about both the learner’s work and the importance of the assessment criteria within the community. In our pilot course using the grade book this semester, each time a rater provided feedback with the rubric, for each dimension of the rubric the rater also indicated their perception of the importance of the dimension on a six-point scale. Our observation, for all groups, was that they increased their appreciation of the value of each rubric dimension, and that as a group, the range in the ratings narrowed. The figure below is representative of the change.
Figure 1. Change in industry raters’ perception (N=6) of the value of rubric dimension 3 (OWN perspective, hypothesis or position is developed and communicated) over the course of the semester.
These changes were brought about through the use of the rubric to assess student work, and not by any conversation about the value of the rubric outside of its immediate application. We imagine that a community of practice would want to develop more explicit mechanisms to talk about, norm on, and refine rubrics as the community gained more experience with the tool.
This is one in a series of posts of material being prepared for presentation for the AAC&U conference in January 2009.
To summarize findings in those other posts, we found in our pilot course that the faculty raters wrote comments less frequently than industry raters, wrote shorter comments, and the comments they wrote were more abstract and de-contextualized from the context of the assigned problem. This is interesting because both groups recognized the growth of the students as measured by the rubric and as measured by an overall assessment of employability, figure below.
Figure 1. Ratings by Industry experts on the employability of student teams based on industry assessments of the students’ projects. The chart shows the assessment at midterm and again at the final. This increase in employability parallels industry’s assessment of student gains on each of the dimensions of the critical thinking rubric.
Next Steps/ Future Work
The implementation of the Harvesting Gradebook used in this pilot course was labor intensive and not practical to scale up beyond a single pilot course. We have imagined a design that would make the process self-service for the faculty member and students [link]. Prior to implementing that software solution, we are exploring developing semi-automated procedures that will make working with multiple pilot courses practical and enable us continue this research.
In response to an earlier post regarding online course eveluation response rate considerations, Jenny Franklin noted the importance of high response rates in small courses because of small sample size. I thought it would be interesting to look at the response rate vs course size distribution. In a college where the Assistant Dean has been a proponent of online evaluations, and has worked diligently on raising response rates, I gathered the data from 348 courses (the whole college sample in Spring 2008).
The x-axis of this graph was scaled to include only courses with enrollments less than 75. That amounted to trimming off about a dozen courses with higher enrollment, including one 200+ student course. The graph is not meant to refute Jenny’s comment, but to provide a picture of the actual experience at WSU. If one were to hypothesize that this wide distribution is likely (and might be difficult to alter), what strategy might one take to course evaluations?
CTLT has been developing a new online survey tool to meet our needs to administer online course evaluations on a large scale. This work is funded in part by a FIPSE grant (BeTA) in collaboration with the TLT Group. One of the by-products we have found in the system we are calling “Survey to make a survey.”
The general idea is that one survey is used to gather information needed to create a second survey and that the data from the first survey can be moved into the second survey by simple, mostly automatic processes: the data in an Excel report coming from the first survey is transformed using Excel functions to be the data needed to create a variant of the survey for a group of respondents in the second survey.
Our first implementation of this idea is a survey exploring instructors’ (largely informal) mid-course formative assessments. The goal was to learn about good practices faculty employ, offer opportunities to learn about other practices, AND, learn which faculty would like to have a formal mid-term course evaluation administered online.
This “survey to make a survey” technique has many possible applications. In describing Matrix Surveys examples, the TLT Group gives an example in the section “Using a Survey of Faculty to Create a Matrix Survey of their Students”
An even more elaborate application of the technique would be to implement our ideas of a transformed grade book. In that example, there are several surveys that are linked to one another and that are used to report results to multiple audiences. This might be done in the semi-automatic means described above, but for uses on a large scale, the linkages enabling survey creation will require greater automation.
George Siemens interviewed Dave Cormier on his recent article in Innovate, Rhizomic Education.
I found this 3 minute portion of the interview more lucid than the article itself regarding community, curriculum and learning in context. It argues, “people will learn by being part of a community (that has a purpose).”
In the interview, Cormier speaks of the course he taught summer 08 – Educational Technology and the Adult Learner. He describes his approach to the course as crediting the student as having their own ability to create knowledge. His premise is that all that is to be known is out there and the learner’s task is to contextualize what is known to the learner and the learner’s needs.
Cormier describes the product of the course as a “textbook” written by the students, but more than an object, he calls it a set of skills and knowledge that students would build out in their own lives (i.e., their professional practice).
I wanted to explore the learning outcomes of the course at greater depth. In his day-8 reflection Cormier says: “What we did not focus on was outlining the ‘takeaways’ that students needed to bring out of the course itself, at least, not in a communal sense. … In a very real sense, each of those students will be taking a very different set of takeaways from this course, related to what they themselves put in, how they contributed to the community and where they are going to take those new literacies when they go back to their own professional practice.”
This didn’t really satisfy me. Take aways are fine, but the students also needed to know where they sit within the community’s norms, the profession’s norms. In following the ideas in Stephen Downes’ Open Source Assessment I think Cormier should have provided the assessment used in the course for community inspection. If I understand Downes’ premise, what is missing in Cormier’s course is the assessment criteria. It is especially important in Cormier’s course because the course-as-community is developing the curriculum — the learners need to be in conversation with the wider community about norms of performance.
With the assessment criteria, you and I could look at the students’ work to see if we could “recognize” (Downes’ term) quality learning outcomes in the products of Cormier’s course. I’m not interested in Cormier ranking the students as much as answering for myself “are they competent?”, “would I hire them?”, “or where are they making progress to those goals?” Those are questions that have an authenticity to a community of practice, such as Cormier seems to be describing.
At WSU we have been thinking about community roles in shaping the assessment criteria for student work and how that changes the ways we think about the grade book. Among those ideas is that the assessment is more than a single number in a gradebook: it might be useful to express numbers along several dimensions of skill, its criteria should be open to community discussion, and the assessment should do more than measure the student, it should inform the student, assignment, course, academic program, and community (that is, it should be a transformative assessment). Consequently, we have been thinking about how the community would have access to the assessment criteria for two purposes (beyond assessing the students): 1) assess the assignment — is this assignment driving at producing the competencies that it holds the students to and, broader, 2) is this assessment measuring the competencies that the community values. Its in this latter sense that Downes’ Open Source Assessment piece is important to our thinking – beyond community as curriculum to community as assessment.
