Different conversations among course participant groups

This is one in a series of posts of material being prepared for presentation for the AAC&U conference in January 2009.

In other posts we have reported on a pilot course using the harvesting gradebook where we observe the assessment success of the tool in both describing student strengths/weaknesses in critical thinking and providing feedback that led to improved performance, and the differences in the behaviors among three groups of raters (student peer, faculty and industry). To continue our examination of the conversation within this community of practice we examined the content of the comments written by the three groups, and also considered the content of comments written by students in self-assessments.

In this course, the recommended procedure for writing comments was to copy phrases from the criteria of the rubric and paste into the comment box, and then elaborate with specific examples or suggestions. Consequently, if reviewers followed the recommended procedure it should produce a high frequency of words in the rubric and tend to focus the conversation on the terminology of the rubric. (A separate study examined the perception of the utility of the rubrics among each group of raters.)

Tag clouds were created using all the words written by each group across all commenting opportunities. Larger and bolder words were used more frequently. As expected from the procedure, language of critical thinking was strongly represented in the comments.

Figure 1. Tag cloud of the most frequently used 50 words of 827 distinct words (omitting insignificant common words such as ‘and’ and ‘the’). This cloud was developed from the comments written in self-assessments of the students.

Figure 2. Tag clouds drawn from the comments made by faculty (N=7) and industry (N=6) groups of raters. Highlight rectangles were added to call out words that are not in the critical thinking rubric, and thus were original text added in comments. “Bag” is an artifact of the assignment being discussed, which was the market forecasting for a high fashion hand bag. Faint numbers give the frequency counts for each word.

We note differences in the language used by faculty and industry. Faculty had the word “problems” as a prominent word but no presence of the word  “problem”. Industry had “problem” as a prominent word but no evidence of “problems” in their tag cloud. (The software treats these words as distinct.)

The word “problems,” in the sense of  “you have problems” appears in Dimension 7, Communication, of the rubric. The word “problem” in the sense of “addressing a problem,” appears in Dimension 1, Problem Identification, of the rubric. Examination of the actual text comments showed that faculty were focused on Dimension 7 “problems” and Industry was focused on Dimension 1 “problem”.

Examples

Faculty:

Few problems with other components of presentation

Industry:

They tell us what they are going to do, but they are not setting up, and are not identifying a problem!  [I wonder why]

The last paragraph of the target market section should have been their summary and that would’ve proved their problem and need for this product.

The word “Market” is in the tag cloud of the industry raters. It is not present at all in the tag cloud of the faculty.  The words “perspectives,” “problem,” and “data” are also prominent in industry language yet absent in the language of faculty.  We conclude Industry professionals work in a different context than faculty (solving a problem rather than grading problems in student work). The following sentences, made out of the words that are prominent in the language of the one group and absent in the other group, illustrate the point:

Industry says:

Designing a “bag” is a “problem” that requires the use of “data” to understand the various “perspectives” present in the “market”.

Faculty says:

“Presents” “views” and draws “conclusions” about an “issue” using “evidence”.