You are currently browsing the tag archive for the 'conversation' tag.
This is one in a series of posts of material being prepared for presentation for the AAC&U conference in January 2009.
In July we postulated a new type of grade book that would have more potential for learners and the community of practice to converse about both the learner’s work and the importance of the assessment criteria within the community. In our pilot course using the grade book this semester, each time a rater provided feedback with the rubric, for each dimension of the rubric the rater also indicated their perception of the importance of the dimension on a six-point scale. Our observation, for all groups, was that they increased their appreciation of the value of each rubric dimension, and that as a group, the range in the ratings narrowed. The figure below is representative of the change.
Figure 1. Change in industry raters’ perception (N=6) of the value of rubric dimension 3 (OWN perspective, hypothesis or position is developed and communicated) over the course of the semester.
These changes were brought about through the use of the rubric to assess student work, and not by any conversation about the value of the rubric outside of its immediate application. We imagine that a community of practice would want to develop more explicit mechanisms to talk about, norm on, and refine rubrics as the community gained more experience with the tool.
This is one in a series of posts of material being prepared for presentation for the AAC&U conference in January 2009.
In July we postulated a new type of grade book that would have more potential for learners and the community of practice to converse about both the learner’s work and the importance of the assessment criteria within the community. From previous experience with an online rubric assessment we believed raters would provide rich textual feedback in addition to numeric ratings and our experience was borne out in this pilot course during Fall 2008.
The rating process involved examining an electronic poster and then completing an online survey form. The survey contained a rubric for rating the work on seven critical thinking dimensions and an option within each rubric dimension to comment about the rating. Raters were coached to copy language from the dimensions criteria and then to elaborate on it with specific examples or suggestions for improvement.
Figure 1. For each group the percent of times that an opportunity was taken to provide textual feedback. Faculty were drawn from within the program, but do not include the course instructor; N=7. Peers are members of other teams reviewing the team’ work; N=84 peers. Professionals are industry domain experts invited to participate; N= 6 professionals. Each reviewer had multiple opportunities to comment, the data above are the aggregate of all the comment opportunities.
The variation in frequency of commenting (Figure 1) led us to examine the quantity of text per comment to assess if the difference among the groups was significant. We observed (Figure 2) that the faculty group was more likely than the other two groups to write short comments, paralleling the faculty proclivity to not comment at all.
Figure 2. The total number of words written per opportunity to comment (one comment opportunity per rubric dimension, 6 opportunities overall for per rating) by each of three groups (invited faculty, invited industry, and student peers). Comment length was lumped into bands (e.g., 0-9 words, 10-19 words). The numbers of comments written by each group were normalized to allow inter-group comparison.
The faculty group took fewer opportunities to comment and wrote shorter comments than either the student peers or industry experts and an examination of the comments confirmed that the faculty took a more summative approach to the process than the other groups.
This behavior on the part of faculty might be understood as an adaption to cope with assessing large numbers of students. What is important about the process is that, because of the use of student peer and industry experts, the challenge of giving feedback is more scalable than in the case where the faculty is the sole assessor. While in this class the students were not well normed with faculty or industry raters, we have other evidence to believe that students can readily norm with instructors and therefore the use of student raters is not a case of “the blind leading the blind.”
Further we have reason to suspect that the students learned by virtue of rating with the rubric and giving peers formative feedback, even if their ratings were more generous than experts might give.
This is one in a series of posts of material being prepared for presentation for the AAC&U conference in January 2009.
In other posts we have reported on a pilot course using the harvesting gradebook where we observe the assessment success of the tool in both describing student strengths/weaknesses in critical thinking and providing feedback that led to improved performance, and the differences in the behaviors among three groups of raters (student peer, faculty and industry). To continue our examination of the conversation within this community of practice we examined the content of the comments written by the three groups, and also considered the content of comments written by students in self-assessments.
In this course, the recommended procedure for writing comments was to copy phrases from the criteria of the rubric and paste into the comment box, and then elaborate with specific examples or suggestions. Consequently, if reviewers followed the recommended procedure it should produce a high frequency of words in the rubric and tend to focus the conversation on the terminology of the rubric. (A separate study examined the perception of the utility of the rubrics among each group of raters.)
Tag clouds were created using all the words written by each group across all commenting opportunities. Larger and bolder words were used more frequently. As expected from the procedure, language of critical thinking was strongly represented in the comments.
Figure 1. Tag cloud of the most frequently used 50 words of 827 distinct words (omitting insignificant common words such as ‘and’ and ‘the’). This cloud was developed from the comments written in self-assessments of the students.
Figure 2. Tag clouds drawn from the comments made by faculty (N=7) and industry (N=6) groups of raters. Highlight rectangles were added to call out words that are not in the critical thinking rubric, and thus were original text added in comments. “Bag” is an artifact of the assignment being discussed, which was the market forecasting for a high fashion hand bag. Faint numbers give the frequency counts for each word.
We note differences in the language used by faculty and industry. Faculty had the word “problems” as a prominent word but no presence of the word “problem”. Industry had “problem” as a prominent word but no evidence of “problems” in their tag cloud. (The software treats these words as distinct.)
The word “problems,” in the sense of “you have problems” appears in Dimension 7, Communication, of the rubric. The word “problem” in the sense of “addressing a problem,” appears in Dimension 1, Problem Identification, of the rubric. Examination of the actual text comments showed that faculty were focused on Dimension 7 “problems” and Industry was focused on Dimension 1 “problem”.
Examples
Faculty:
Few problems with other components of presentation
Industry:
They tell us what they are going to do, but they are not setting up, and are not identifying a problem! [I wonder why]
The last paragraph of the target market section should have been their summary and that would’ve proved their problem and need for this product.
The word “Market” is in the tag cloud of the industry raters. It is not present at all in the tag cloud of the faculty. The words “perspectives,” “problem,” and “data” are also prominent in industry language yet absent in the language of faculty. We conclude Industry professionals work in a different context than faculty (solving a problem rather than grading problems in student work). The following sentences, made out of the words that are prominent in the language of the one group and absent in the other group, illustrate the point:
Industry says:
Designing a “bag” is a “problem” that requires the use of “data” to understand the various “perspectives” present in the “market”.
Faculty says:
“Presents” “views” and draws “conclusions” about an “issue” using “evidence”.
