You are currently browsing the tag archive for the 'aacu' tag.
AAC&U Conference , April 2-4 2009
Round Table presentation
Nils S. Peterson, Assistant Director of The Center for Teaching, Learning, and Technology—Washington State University
Abstract: Washington State University has explored SharePoint (MOSS) as a lightweight learning management system (LMS) and SharePoint Mysites as an ePortfolio platform, supplemented with social networking tools and strategies. Integration between course spaces and portfolios has been done in a “hub and spoke” model. New strategies for facilitating and assessing learning necessitate a substantial change in faculty roles. In this session, participants will (1) explore the critical and integrative thinking skills that students will need in 21st century Web 2.0 learning/work environments and (2) use this exploration to reflect on novel assignments and faculty roles needed to (3) facilitate this learning.
The following documents were part of this discussion:
- Introduction, activity, and readings
- Current description of WSU’s work (see also this blog post from AAC&U Jan. 2009)
- Updated Learning Spectrum self-assessment
- Four strategies (different points on the Learning Spectrum)
CTLT has been thinking about portfolios for learning and their relationship to institutionally supported learning tools and course designs. This thinking has us moving away from the traditional LMS. In a February 2008 Campus Technology interview, Gary Brown introduced the term “harvesting gradebook” to describe the gradebook that faculty need to work in these decentralized environments. As originally articulated by Gary, the gradebook “harvested” student work, storing copies of the work within itself where it was assessed.
On further discussion, the concept became inverted, what was “harvested” were assessments, from work that remained in-situ.
This inversion of the idea allowed the widening of the community that could be involved in the assessment. There are ways that the instructor, as well as the program can learn from this transformed idea about a gradebook that are responsive to course and program improvements, as well as useful in accreditation.
A pilot course using these ideas earned the NUTN 2009 Best Resaerch Paper award. Here is the video made for the award ceremony.
At the AAC&U conference in Seattle, Jan 22-24, we presented these ideas at a round table on ePortfolios Friday morning. (Authentic Assessment of Learning in Global Contexts) Nils Peterson, Gary Brown, Jayme Jacobson, Theron DesRosier
In February 2009, a Campus Technology article summarized a pilot offering of a course that used this latter harvesting of assessments model beginning to demonstrate how a community could effectively participate in the process.
This post serves as a table of contents to materials from our “Authentic assessment of learning in global contexts” AAC&U presentation and background to the story in Campus Technology.
- Abstract submitted to AAC&U for the session
- The 10 point self-assessment chart handed out at AAC&U
- The booklet that was handed out (below are topics in the booklet)
- Nature of the community feedback
- WSU 2007-08 ePortfolio Contest (call to contestants)
- ePortfolio Contest Results (where important lessons were learned)
- Harvesting (Transformative) Gradebook
- Assumptions behind the idea
- Overview (synthesis and reflection at end of summer 08)
- Inception of idea
- Demonstration of Feasibility of Implementation (using SharePoint and Google Docs)
- Radar graphs for multidimensional data (description)
- Scaling up the Implementation (design)
- Pilot course (Fall 08)
- Assumptions behind the idea
- Getting the assessment embedded
- What the gradebook looks like
- The conversation among the community
- Results of the pilot course (impact on learners)
In July we made a proposal for a presentation for the AAC&U conference in January 2009 (which has been accepted, see you in Seattle). A previous post serves as an index to the materials we are collecting for that presentation. The AAC&U call for proposals asked a series of questions that reveal their perspective on the roles of the institution, faculty, students and the wider community in the learning enterprise.
The assumptions and perspectives that appear to underlie AAC&U’s framing of the questions are summarized in the table below. The preamble to the call for proposals also expresses a concern for maintaining the relevance of colleges and universities and maintaining the importance of the values of a liberal education. We see a conflict between the AAC&U assumptions and the goal of maintaining relevance in a Web 2.0 world.
The work we are reporting at the conference comes from a different set of assumptions about the learning enterprise. The perspective is community centered. It uses a set of assumptions that have previously existed within institutions of higher learning, and to some extent, still exist at the PhD level, but which seem to have been displaced by the challenges of increasing scale and decreasing resources. We are exploring ways in which the Internet, and particularly the Web 2.0 perspective of the Internet, can facilitate a transformation to a different set of assumptions.
Central to this thinking is our work from last summer on a Harvesting (or Transformative) Grade book. An important aspect of this re-visioning of the grade book is its move into a Web 2.0 context and its role in facilitating our assumptions about learning in communities (right column in table). This work is related to ideas Stephen Downes shared about group vs network learning and the lists below are worth examining in light of his diagram of that idea. This little video about connectivism gives some other ideas about faculty roles and implementation of our community-centric assumptions. Some very good comments about the role of the faculty appears ¾ of the way into the video.
| Institution-centric (AAC&U) view | Community-centric View |
| The institution and faculty are the central node. | The learner is the central node; learners include all members of the discourse community (or Community of Practice). |
| The faculty member is the grading authority. | Community of Practice holds responsibility (and the social capital) for assessment, and assessment (from the root “assay” as in to sit with) is most useful when it is formative, when it is understood to be constructive feedback rather than an authority’s judgment. |
| Faculty define the assessment criteria. | Community of Practice, by virtue of expert consensus, validates the assessment instrument. |
| Feedback to the student about learning criteria is masked by the letter grade. | Students merit feedback direct and unfiltered from the community using criteria that the community articulates. |
| Institution is the established (credentialing) authority. | Community of Practice is the implicit credentialing authority; the university is the facilitator of that credentialing and of community building. |
| The unit of analysis is the course, bounded in time, content, and brick or virtual (LMS) space. | The unit of analysis is the problem, problems are not bound in time or content. |
| Problems for students to study are (artificially) constrained by course and disciplinary boundaries. | Communities identify authentic problems that are interdisciplinary and reach beyond the definition of the course. |
| Students rarely share their work with, or receive feedback from, public audiences. | ePortfolios built over multiple year are learner owned (not university owned) and used to communicate (collect, select, connect, project, and reflect) with and get feedback from wider communities. |
| Faculty are a gatekeeper between the students and the community of practice. | Students are anticipated to join communities of practice, faculty may introduce students to the community. |
| Faculty have all the social capital within both the classroom and the community of practice. | (Student) learning is social and therefore learners are building social capital within the classroom and withing the community of practice. |
Related posts
1. Pilot course using the Harvesting Gradebook
2. Rich assessment data from a Harvesting Gradebook
3. Community conversation about assessment criteria
4. Differences in engaging the discussion among participant groups
5. Different conversations among participant groups
6. Evidence for Impact on Learning
First experience
This is one in a series of posts of material being prepared for presentation at the AAC&U conference in January 2009
The examples in the following posts are all drawn from a course offered at Washington State University in Fall 2008. We have previously demonstrated implementations of the gradebook, but none were practical for course use. In the course described below, we used our online survey tool, Skylight, to create an implementation that (while still labor intensive) was feasible for a pilot. We have ideas for a more robust implementation of the gradebook process that would make the process scalable to enterprise levels.
The course involved 87 students in 14 teams working on a semester-long design project. Each team created an electronic poster presentation (see poster template) of their work at midterm for formative feedback and then created a revised electronic poster as a final project. The instructor recruited industry professionals and other instructors to give feedback to the posters using a rubric derived from WSU’s critical and integrative thinking rubric. Students used the rubric for self-evaluation and also to give peer feedback.
Students were able to see results of the feedback in real time, via a dashboard we adapted from a tool WSU uses to returning course evaluation results to instructors.
Student teams were each working in team blogs (Blogger) where they compiled research information and other materials required for the course project. The instructor and other faculty, and the industry experts had access to the blogs. The project posters were placed in a blog post, along with a URL to the feedback survey (this was the data entry mechanism for the Harvesting Gradebook).
The instructor, industry experts, and peers from other teams gave feedback using the feedback survey at mid-term. Student teams were able to use this information to revise their work, which was then reviewed by 7 faculty from the program, 6 industry experts, and peers.
The following reports will demonstrate that the Harvesting Gradebook can serve as an assessment tool capable of distinguishing among student performance (as hypothesized last summer) and that it is a tool that can facilitate a conversation among stakeholders about the criteria used in the assessment, in addition to a conversation about the work being assessed. The former we contend are important elements for a community’s discussion of the learning of its participants.
This is one in a series of posts of material being prepared for presentation for the AAC&U conference in January 2009.
In July we postulated a new type of grade book that would have more potential for learners and learning than the traditional grade book does because it can open up richer forms of assessment. Included in that description was an illustration of the grade book, showing that it was conceivable student performances could be rendered in ways that would be both descriptive of students learning strengths and deficits and also informative to instructors and programs about the efficacy of the assignments.
In our pilot course this semester, we were able to gather data that confirms one of our hypotheses from the summer – student work can be rendered in descriptive ways and with enough resolution to tell students apart by their various abilities. The tables below compare the self- and industry ratings, and self- and faculty ratings on the seven dimensions of the rubric used. And also shows the hiring decision (a holistic judgment) by the self-, faculty- and industry- raters.
Figure 1. Student Team 1 assessment data from all seven dimensions of the rubric, comparing self-assessment to industry and to faculty assessment. In each of the figures above, the rubric dimensions are as follows: 1. Ability to identify the problem; 2. Ability to consider context; 3. OWN perspective, hypothesis or position is developed and communicated; 4. Ability to select, evaluate and interpret data; 5. Integration of OTHER (disciplinary) perspectives and positions; 6. Identifies and assesses conclusions, implications, and consequences; 7. Communicates effectively. For more information on the radar plot see this.
Figure 2. Student Team 2 assessment data from all seven dimensions of the rubric, comparing self-assessment to industry and to faculty assessment. Rubric dimensions are the same as figure 1.
Reporting the data to students
The system provided each student team real-time access to their assessments using a “dashboard” that students could log into to retrieve reports from the rubric surveys. The dashboard was originally designed to return online course evaluation results to faculty. This tool is analogous in function to the online gradebook in a learning management system, in that it shows a student one row of grades and the instructor a row for each student.
Analysis of the data
No activities to norm with the rubric were included with the class procedures. The lack of norming is evident in the data. While lack of norming of raters is a limitation of this pilot, within groups (students, faculty, and industry) raters are fairly well normed. The lack of intra-group norming might be seen as an occasion for conversation. Faculty and industry are in closer agreement as groups, but faculty tended to be a bit more generous.
The figures above also show the hiring decision (an aggregate assessment) made by each rater. Industry and faculty ratings on the rubric (below competency) correspond to their employability assessments, internship or less.
Comparing midterm to final there is evidence of growth in the eyes of industry and faculty, though the growth is in different directions and at different rates for each student team.
It is worth noting, students are over-confident of their abilities, consistent with their novice role within the community of practice. Most students also reflected their self-perception of learning growth during the course as increasing their competence. In a few cases, there was evidence that as students learned during the course they reduced their self-assessments, moving more in line with the assessments of industry and faculty.
This is one in a series of posts of material being prepared for presentation for the AAC&U conference in January 2009.
In July we postulated a new type of grade book that would have more potential for learners and the community of practice to converse about both the learner’s work and the importance of the assessment criteria within the community. In our pilot course using the grade book this semester, each time a rater provided feedback with the rubric, for each dimension of the rubric the rater also indicated their perception of the importance of the dimension on a six-point scale. Our observation, for all groups, was that they increased their appreciation of the value of each rubric dimension, and that as a group, the range in the ratings narrowed. The figure below is representative of the change.
Figure 1. Change in industry raters’ perception (N=6) of the value of rubric dimension 3 (OWN perspective, hypothesis or position is developed and communicated) over the course of the semester.
These changes were brought about through the use of the rubric to assess student work, and not by any conversation about the value of the rubric outside of its immediate application. We imagine that a community of practice would want to develop more explicit mechanisms to talk about, norm on, and refine rubrics as the community gained more experience with the tool.
This is one in a series of posts of material being prepared for presentation for the AAC&U conference in January 2009.
In July we postulated a new type of grade book that would have more potential for learners and the community of practice to converse about both the learner’s work and the importance of the assessment criteria within the community. From previous experience with an online rubric assessment we believed raters would provide rich textual feedback in addition to numeric ratings and our experience was borne out in this pilot course during Fall 2008.
The rating process involved examining an electronic poster and then completing an online survey form. The survey contained a rubric for rating the work on seven critical thinking dimensions and an option within each rubric dimension to comment about the rating. Raters were coached to copy language from the dimensions criteria and then to elaborate on it with specific examples or suggestions for improvement.
Figure 1. For each group the percent of times that an opportunity was taken to provide textual feedback. Faculty were drawn from within the program, but do not include the course instructor; N=7. Peers are members of other teams reviewing the team’ work; N=84 peers. Professionals are industry domain experts invited to participate; N= 6 professionals. Each reviewer had multiple opportunities to comment, the data above are the aggregate of all the comment opportunities.
The variation in frequency of commenting (Figure 1) led us to examine the quantity of text per comment to assess if the difference among the groups was significant. We observed (Figure 2) that the faculty group was more likely than the other two groups to write short comments, paralleling the faculty proclivity to not comment at all.
Figure 2. The total number of words written per opportunity to comment (one comment opportunity per rubric dimension, 6 opportunities overall for per rating) by each of three groups (invited faculty, invited industry, and student peers). Comment length was lumped into bands (e.g., 0-9 words, 10-19 words). The numbers of comments written by each group were normalized to allow inter-group comparison.
The faculty group took fewer opportunities to comment and wrote shorter comments than either the student peers or industry experts and an examination of the comments confirmed that the faculty took a more summative approach to the process than the other groups.
This behavior on the part of faculty might be understood as an adaption to cope with assessing large numbers of students. What is important about the process is that, because of the use of student peer and industry experts, the challenge of giving feedback is more scalable than in the case where the faculty is the sole assessor. While in this class the students were not well normed with faculty or industry raters, we have other evidence to believe that students can readily norm with instructors and therefore the use of student raters is not a case of “the blind leading the blind.”
Further we have reason to suspect that the students learned by virtue of rating with the rubric and giving peers formative feedback, even if their ratings were more generous than experts might give.
This is one in a series of posts of material being prepared for presentation for the AAC&U conference in January 2009.
In other posts we have reported on a pilot course using the harvesting gradebook where we observe the assessment success of the tool in both describing student strengths/weaknesses in critical thinking and providing feedback that led to improved performance, and the differences in the behaviors among three groups of raters (student peer, faculty and industry). To continue our examination of the conversation within this community of practice we examined the content of the comments written by the three groups, and also considered the content of comments written by students in self-assessments.
In this course, the recommended procedure for writing comments was to copy phrases from the criteria of the rubric and paste into the comment box, and then elaborate with specific examples or suggestions. Consequently, if reviewers followed the recommended procedure it should produce a high frequency of words in the rubric and tend to focus the conversation on the terminology of the rubric. (A separate study examined the perception of the utility of the rubrics among each group of raters.)
Tag clouds were created using all the words written by each group across all commenting opportunities. Larger and bolder words were used more frequently. As expected from the procedure, language of critical thinking was strongly represented in the comments.
Figure 1. Tag cloud of the most frequently used 50 words of 827 distinct words (omitting insignificant common words such as ‘and’ and ‘the’). This cloud was developed from the comments written in self-assessments of the students.
Figure 2. Tag clouds drawn from the comments made by faculty (N=7) and industry (N=6) groups of raters. Highlight rectangles were added to call out words that are not in the critical thinking rubric, and thus were original text added in comments. “Bag” is an artifact of the assignment being discussed, which was the market forecasting for a high fashion hand bag. Faint numbers give the frequency counts for each word.
We note differences in the language used by faculty and industry. Faculty had the word “problems” as a prominent word but no presence of the word “problem”. Industry had “problem” as a prominent word but no evidence of “problems” in their tag cloud. (The software treats these words as distinct.)
The word “problems,” in the sense of “you have problems” appears in Dimension 7, Communication, of the rubric. The word “problem” in the sense of “addressing a problem,” appears in Dimension 1, Problem Identification, of the rubric. Examination of the actual text comments showed that faculty were focused on Dimension 7 “problems” and Industry was focused on Dimension 1 “problem”.
Examples
Faculty:
Few problems with other components of presentation
Industry:
They tell us what they are going to do, but they are not setting up, and are not identifying a problem! [I wonder why]
The last paragraph of the target market section should have been their summary and that would’ve proved their problem and need for this product.
The word “Market” is in the tag cloud of the industry raters. It is not present at all in the tag cloud of the faculty. The words “perspectives,” “problem,” and “data” are also prominent in industry language yet absent in the language of faculty. We conclude Industry professionals work in a different context than faculty (solving a problem rather than grading problems in student work). The following sentences, made out of the words that are prominent in the language of the one group and absent in the other group, illustrate the point:
Industry says:
Designing a “bag” is a “problem” that requires the use of “data” to understand the various “perspectives” present in the “market”.
Faculty says:
“Presents” “views” and draws “conclusions” about an “issue” using “evidence”.
This is one in a series of posts of material being prepared for presentation for the AAC&U conference in January 2009.
To summarize findings in those other posts, we found in our pilot course that the faculty raters wrote comments less frequently than industry raters, wrote shorter comments, and the comments they wrote were more abstract and de-contextualized from the context of the assigned problem. This is interesting because both groups recognized the growth of the students as measured by the rubric and as measured by an overall assessment of employability, figure below.
Figure 1. Ratings by Industry experts on the employability of student teams based on industry assessments of the students’ projects. The chart shows the assessment at midterm and again at the final. This increase in employability parallels industry’s assessment of student gains on each of the dimensions of the critical thinking rubric.
Next Steps/ Future Work
The implementation of the Harvesting Gradebook used in this pilot course was labor intensive and not practical to scale up beyond a single pilot course. We have imagined a design that would make the process self-service for the faculty member and students [link]. Prior to implementing that software solution, we are exploring developing semi-automated procedures that will make working with multiple pilot courses practical and enable us continue this research.
Nils Peterson, Gary Brown, Jayme Jacobson, Theron DesRosier
The AAC&U 2009 conference, Global Challenge, College Learning, and America’s Promise asks a series of questions about “the relationship between college learning and society” and the Call for Proposal notes “[College and university] failure to fully engage our publics with the kinds of learning best suited to our current needs…” [For further evidence of this failure, see Boeing to Rank Colleges, Chronicle, 9/19/08 (comment added 9/18/08)]
[Additional evidence of this challenge comes from the students themselves, in this manifesto Innovate or Die, Student2oh.org, accessed 10/7/08, Anthony Chiveta argues that this generation of students understands the Web 2.0 concept and wants it applied to education (see also Learning 2.0) (comment added 10/7/08)]
The following is an abstract submitted in response to the Call above. It served as an opportunity to “rise above” our work on these topics and reflect on its implications.
—
Assessment of learning by a community inside and outside the classroom is a key component to developing students’ global competencies and building a strong relationship between college learning and society.
In 2007-08 Washington State University conducted an ePortfolio Contest to demonstrate ways to harness the interests and expertise of the WSU community to address real world problems encountered by communities both locally and globally. It called upon contestants to collaborate with community members – institutional, local, or global – to identify a problem, explore solutions, develop a plan, and then take steps toward implementing that plan. Contestants were asked to use electronic portfolios to capture and reflect on their collaborative problem-solving processes and the impact of their projects. Judges from industry, the local community, and WSU used a rubric based on the WSU Critical Thinking Rubric to evaluate the portfolios. Since the contest, we have been distilling design principles for portfolios that facilitate learning.
This exploration has taught us to value the learner consciously leaving a ‘learning trace’ as they work on a problem, and we believe the capturing and sharing of that trace is an important part of documenting learning. A striking example illustrating a learning trace in a portfolio is one of the winners in our contest. Not only does this portfolio exhibit a learning trace, it captures feedback from the community regarding the quality of the work. A recent AAC&U survey of employers supports this bias for richer documentation of learner skills.
Our thinking about portfolios for learning is moving us away from traditional ideas about courses contained in classrooms and toward Stephen Downes’ eLearning 2.0 ideas: “Students’ [learning portfolios] are often about something from their own range of interests, rather than on a course topic or assigned project. More importantly, what happens when students [work in this way], is that a network of interactions forms-much like a social network, and much like Wenger’s community of practice.” And far from being trivial social networking, our portfolio contest captured rich and substantive learning happening in the community outside the classroom.
But documentation of learning, without feedback, guidance and assessment leaves the learner to work without the support of a community, which has led us to a recognition that grade books are QWERTY artifacts of a Learning 1.0 model. To address that, we have been exploring ways to transform the grade book (see this and this) to support learners working simultaneously within the university and within their communities of practice. This approach to a grade book has the additional benefit for the scholarship of teaching and learning of gathering feedback on the assignment, course and program from the community at the same time that it invites the community to assess a learner’s work. Such feedback can help the university engage its publics in a discussion of the kinds of learning most suited to current needs.
The questions in the AAC&U Call will be used to help the audience highlight and frame the discussion of our ideas.
—
The Call “invites proposals of substantive, engaging sessions that will raise provocative questions, that will engage participants in discussion, and that will create and encourage dialogue–before, during, and after the conference itself.”
In the spirit of the AAC&U call, you are invited to engage this discussion before or after the conference by posting comments here, on the related pages, or by tracking back from your own blog, and then meet us in Seattle to further the conversation.
