You are currently browsing the tag archive for the 'feedback' tag.
In July we made a proposal for a presentation for the AAC&U conference in January 2009 (which has been accepted, see you in Seattle). A previous post serves as an index to the materials we are collecting for that presentation. The AAC&U call for proposals asked a series of questions that reveal their perspective on the roles of the institution, faculty, students and the wider community in the learning enterprise.
The assumptions and perspectives that appear to underlie AAC&U’s framing of the questions are summarized in the table below. The preamble to the call for proposals also expresses a concern for maintaining the relevance of colleges and universities and maintaining the importance of the values of a liberal education. We see a conflict between the AAC&U assumptions and the goal of maintaining relevance in a Web 2.0 world.
The work we are reporting at the conference comes from a different set of assumptions about the learning enterprise. The perspective is community centered. It uses a set of assumptions that have previously existed within institutions of higher learning, and to some extent, still exist at the PhD level, but which seem to have been displaced by the challenges of increasing scale and decreasing resources. We are exploring ways in which the Internet, and particularly the Web 2.0 perspective of the Internet, can facilitate a transformation to a different set of assumptions.
Central to this thinking is our work from last summer on a Harvesting (or Transformative) Grade book. An important aspect of this re-visioning of the grade book is its move into a Web 2.0 context and its role in facilitating our assumptions about learning in communities (right column in table). This work is related to ideas Stephen Downes shared about group vs network learning and the lists below are worth examining in light of his diagram of that idea. This little video about connectivism gives some other ideas about faculty roles and implementation of our community-centric assumptions. Some very good comments about the role of the faculty appears ¾ of the way into the video.
| Institution-centric (AAC&U) view | Community-centric View |
| The institution and faculty are the central node. | The learner is the central node; learners include all members of the discourse community (or Community of Practice). |
| The faculty member is the grading authority. | Community of Practice holds responsibility (and the social capital) for assessment, and assessment (from the root “assay” as in to sit with) is most useful when it is formative, when it is understood to be constructive feedback rather than an authority’s judgment. |
| Faculty define the assessment criteria. | Community of Practice, by virtue of expert consensus, validates the assessment instrument. |
| Feedback to the student about learning criteria is masked by the letter grade. | Students merit feedback direct and unfiltered from the community using criteria that the community articulates. |
| Institution is the established (credentialing) authority. | Community of Practice is the implicit credentialing authority; the university is the facilitator of that credentialing and of community building. |
| The unit of analysis is the course, bounded in time, content, and brick or virtual (LMS) space. | The unit of analysis is the problem, problems are not bound in time or content. |
| Problems for students to study are (artificially) constrained by course and disciplinary boundaries. | Communities identify authentic problems that are interdisciplinary and reach beyond the definition of the course. |
| Students rarely share their work with, or receive feedback from, public audiences. | ePortfolios built over multiple year are learner owned (not university owned) and used to communicate (collect, select, connect, project, and reflect) with and get feedback from wider communities. |
| Faculty are a gatekeeper between the students and the community of practice. | Students are anticipated to join communities of practice, faculty may introduce students to the community. |
| Faculty have all the social capital within both the classroom and the community of practice. | (Student) learning is social and therefore learners are building social capital within the classroom and withing the community of practice. |
Related posts
1. Pilot course using the Harvesting Gradebook
2. Rich assessment data from a Harvesting Gradebook
3. Community conversation about assessment criteria
4. Differences in engaging the discussion among participant groups
5. Different conversations among participant groups
6. Evidence for Impact on Learning
First experience
This is one in a series of posts of material being prepared for presentation at the AAC&U conference in January 2009
The examples in the following posts are all drawn from a course offered at Washington State University in Fall 2008. We have previously demonstrated implementations of the gradebook, but none were practical for course use. In the course described below, we used our online survey tool, Skylight, to create an implementation that (while still labor intensive) was feasible for a pilot. We have ideas for a more robust implementation of the gradebook process that would make the process scalable to enterprise levels.
The course involved 87 students in 14 teams working on a semester-long design project. Each team created an electronic poster presentation (see poster template) of their work at midterm for formative feedback and then created a revised electronic poster as a final project. The instructor recruited industry professionals and other instructors to give feedback to the posters using a rubric derived from WSU’s critical and integrative thinking rubric. Students used the rubric for self-evaluation and also to give peer feedback.
Students were able to see results of the feedback in real time, via a dashboard we adapted from a tool WSU uses to returning course evaluation results to instructors.
Student teams were each working in team blogs (Blogger) where they compiled research information and other materials required for the course project. The instructor and other faculty, and the industry experts had access to the blogs. The project posters were placed in a blog post, along with a URL to the feedback survey (this was the data entry mechanism for the Harvesting Gradebook).
The instructor, industry experts, and peers from other teams gave feedback using the feedback survey at mid-term. Student teams were able to use this information to revise their work, which was then reviewed by 7 faculty from the program, 6 industry experts, and peers.
The following reports will demonstrate that the Harvesting Gradebook can serve as an assessment tool capable of distinguishing among student performance (as hypothesized last summer) and that it is a tool that can facilitate a conversation among stakeholders about the criteria used in the assessment, in addition to a conversation about the work being assessed. The former we contend are important elements for a community’s discussion of the learning of its participants.
A grade book traditionally is a one way reporting mechanism-it reports to students their performance as assessed by the instructor who designed the activity. Learning from grades in this impoverished but pervasive model is largely one way-the student learns, presumably, from the professor’s grade. What does a student really learn from a letter or number grade? What does the faculty member learn from this transaction that will help him or her improve? What does a program or institution learn? We are exploring ways to do better.
There is a second goal embedded in this effort. In a Web 2.0 environment, an environment where social networking and learning are ubiquitous like the tools that are supporting that exchange, how might one, be they student, faculty, or organization, harvest that learning, deepen it, and disseminate it more broadly.
Further, and in very practical terms, students are swirling, that is, they are attending one or more institutions simultaneously and several in their educational career. This is happening even as:
- institutions are presuming that retention is of utmost importance,
- standardized testing is being implemented so that consumers of all stripes might compare educational “products” and
- the importance of diversity is finally gaining real support.
The implications of this is deeply challenging on many levels. How can one gather feedback from multiple sources and stakeholders at all levels to enrich our understanding about learning? We are exploring potentially more productive alternatives.
This post is in a series exploring issues related to transforming the grade book. We are seeking to apply the ideas of Transformative Assessment to grade books in order to develop a tool that will help the institution learn and improve at all levels, from students on assignments, to instructors teaching courses, to programs adjusting to meet the needs of communities (graduate schools, employers, society at large).
Much has been written lately about the Collegiate Learning Assessment (CLA), including some significant detractors (NYTimes & Chronicle of Higher Education)
Our interest here is exploring how the data that might be gained from the CLA or from our Transformed Grade book might be useful for the following purposes:
- Students analyze their own performance and growth over time
- Faculty advise students on courses, plans of study, areas for focus
- Faculty analyze the utility of their assignments in facilitating student learning
- Faculty analyze their assignments’ alignment with program goals
- Academic programs collect better profiles of students applying to major
- Academic programs analyze courses relative to program goals
- Community (employers, grad school, society) examine and discuss both the goals and the success of programs
- Demonstration of reliability in each of the assessments above
Figure 1. (click to enlarge) CLA report of results (left) and WSU Transformative Grade Book (right). Information to help read the CLA graph can be found on page 8 of this PDF. Information on reading the WSU multidimensional graph can be found here.
In the graphs above, the CLA data provides no information to the student, and no information to the instructor or program in terms of what/ how to change to alter the CLA graduation score outcome.
The CLA reduces a whole entering or exiting class of students to a single number. This is too much “chunking,” it does not inform individuals, faculty, or programs enough about the learning outcomes or the places improvements can be made to reach them.
Summary
By making the rubric and the assessment process public and using it across multiple assignments and courses, the potential to analyze and respond to the assessment is much greater. While the multidimensional graph shown compares three perspectives on one assignment, it could just as easily compare scores for a student or group recorded at different points in time.
Examples of how a transformative assessment approach can address multiple audiences and needs:
Students analyze their own performance on assignments and over time
The figure below shows a “grade book” where the grades for each assignment are recorded as a multidimensional graph. A student would see only one row, the instructor would see all rows. In this example the later assignments are layered on top of their predecessors for comparisons. (Data for this example are hypothetical, but were created with particular student abilities and stories in mind.)
Figure 2. Multidimensional Transformed Grade Book (click figure to enlarge)
Faculty advise students on courses, plans of study, areas for focus
In the grade book above, imagine that the last column were an average “final grade” across the preceding assignments. It seems clear that students’ different abilities could be seen and these differences used by faculty for advising.
Faculty analyze the utility of their assignments in facilitating student learning
We are proposing that faculty’s assignments, not just student work, be scored with the same rubric (asking “How well does this assignment elicit …” rather than “How well does this student’s work demonstrate …”) In the figure below the Grade book has been augmented to show how each assignment’s average rating and individual ratings from three groups: student, faculty, community. Comparison of assignment rating to student work rating offers a chance to examine how a particular assignment is (or fails to) facilitate student success in a particular dimension.
Figure 3. Using the same rubric, comparison of average student score on 5 assignments (below bar) with the average rating of faculty, student and community on each of the assignments (above bar). (Click figure to enlarge)
This figure shows the transformed grade book with data about the assignment’s ratings paired with the average rating of student work on that assignment. Assignment ratings come from students, faculty peers, and external community.
Faculty analyze their assignments to align better with program goals
Similarly, faculty could compare their assignments and student work to the program’s goals (which is presumably the source of the rubric and the source of the “competency” threshold).
Academic programs collect better profiles of students applying to major
Many professional programs have portfolios or other performance demonstrations for admission to the major and upper division coursework. A student’s portfolio could be scored with a rubric and the resulting multidimensional graph would communicate to the student and program the student’s strengths and weaknesses. This could benefit both screening processes and could communicate to lower division courses that nature of student deficiencies in ways that those programs would know what actions to take.
Academic programs analyze courses relative to program goals
By using the same tool to assess assignments and student products within a course and by having that tool be the definition of the program’s goals, the degree of course alignment with program goals would be readily demonstrated. Further it would be possible to review a series of courses for the ways they cumulatively contribute to program goals.
Community (employers, grad school, society) examine and discuss both the goals and the success of programs
Where courses have assignments that result in a public performance by the student (portfolio, recital, internship, etc) the community can be engaged with assessing the student and with the fit between the program’s expression of its goals and the community’s (perhaps implicit) standard.
Demonstration of reliability in each of the assessments above
One of the challenges in using diverse groups of raters and qualitative schemes such as rubrics, is measuring, maintaining, and demonstrating reliability in among the raters. We have applied the multidimensional graph to compare ratings from multiple judges in our ePortfolio contest. The agreement (or divergence) that the graphs showed correlated well with the amount of discussion and (dis-)agreement when the judges met.
Nils Peterson, Gary Brown, Jayme Jacobson, Theron DesRosier
The AAC&U 2009 conference, Global Challenge, College Learning, and America’s Promise asks a series of questions about “the relationship between college learning and society” and the Call for Proposal notes “[College and university] failure to fully engage our publics with the kinds of learning best suited to our current needs…” [For further evidence of this failure, see Boeing to Rank Colleges, Chronicle, 9/19/08 (comment added 9/18/08)]
[Additional evidence of this challenge comes from the students themselves, in this manifesto Innovate or Die, Student2oh.org, accessed 10/7/08, Anthony Chiveta argues that this generation of students understands the Web 2.0 concept and wants it applied to education (see also Learning 2.0) (comment added 10/7/08)]
The following is an abstract submitted in response to the Call above. It served as an opportunity to “rise above” our work on these topics and reflect on its implications.
—
Assessment of learning by a community inside and outside the classroom is a key component to developing students’ global competencies and building a strong relationship between college learning and society.
In 2007-08 Washington State University conducted an ePortfolio Contest to demonstrate ways to harness the interests and expertise of the WSU community to address real world problems encountered by communities both locally and globally. It called upon contestants to collaborate with community members – institutional, local, or global – to identify a problem, explore solutions, develop a plan, and then take steps toward implementing that plan. Contestants were asked to use electronic portfolios to capture and reflect on their collaborative problem-solving processes and the impact of their projects. Judges from industry, the local community, and WSU used a rubric based on the WSU Critical Thinking Rubric to evaluate the portfolios. Since the contest, we have been distilling design principles for portfolios that facilitate learning.
This exploration has taught us to value the learner consciously leaving a ‘learning trace’ as they work on a problem, and we believe the capturing and sharing of that trace is an important part of documenting learning. A striking example illustrating a learning trace in a portfolio is one of the winners in our contest. Not only does this portfolio exhibit a learning trace, it captures feedback from the community regarding the quality of the work. A recent AAC&U survey of employers supports this bias for richer documentation of learner skills.
Our thinking about portfolios for learning is moving us away from traditional ideas about courses contained in classrooms and toward Stephen Downes’ eLearning 2.0 ideas: “Students’ [learning portfolios] are often about something from their own range of interests, rather than on a course topic or assigned project. More importantly, what happens when students [work in this way], is that a network of interactions forms-much like a social network, and much like Wenger’s community of practice.” And far from being trivial social networking, our portfolio contest captured rich and substantive learning happening in the community outside the classroom.
But documentation of learning, without feedback, guidance and assessment leaves the learner to work without the support of a community, which has led us to a recognition that grade books are QWERTY artifacts of a Learning 1.0 model. To address that, we have been exploring ways to transform the grade book (see this and this) to support learners working simultaneously within the university and within their communities of practice. This approach to a grade book has the additional benefit for the scholarship of teaching and learning of gathering feedback on the assignment, course and program from the community at the same time that it invites the community to assess a learner’s work. Such feedback can help the university engage its publics in a discussion of the kinds of learning most suited to current needs.
The questions in the AAC&U Call will be used to help the audience highlight and frame the discussion of our ideas.
—
The Call “invites proposals of substantive, engaging sessions that will raise provocative questions, that will engage participants in discussion, and that will create and encourage dialogue–before, during, and after the conference itself.”
In the spirit of the AAC&U call, you are invited to engage this discussion before or after the conference by posting comments here, on the related pages, or by tracking back from your own blog, and then meet us in Seattle to further the conversation.
Jayme Jacobson, Theron DesRosier, Nils Peterson with help from Jack Wacknitz
Previously we showed examples of how a transformed grade book (or Harvesting grade book ) might look from the perspective of a piece of student work seeking feedback. That demonstration was hand built and is not practical to scale up to the level of an academic program. Even the smaller scale use such as that suggested by a recent piece in the Chronicle entitled Portfolios Are Replacing Qualifying Exams would benefit from some automation. This post outlines a design for a software application that could plug into the Skylight Matrix Survey System (a new survey tool WSU is developing).
There are two workflows that we have previously described in general terms, one for the instructor to register assignments and another for students to request feedback as they work on those assignments. In the figure below we outline a series of screen shots for the instructor registering the assignment. During the registration process the instructor is matching the assignment to the assessment rubric dimensions that are appropriate. We view this registration of the assessments as a possible implementation of Stephen Downes’ idea in Open Source Assessment.
Instructor Dashboard (click image to enlarge)
This examples shows how an instructor might use a dashboard to register and monitor assignments. The workflow shows capturing the assignment, assigning rubric dimensions to be used for assessing both the student work and the assignment itself and ends with routing the assignment for review. This mock-up does not show how the instructor would see the student work created in response to the assignment, or the scores associated with that student work . The next step in the workflow would require an upload of the assignment so that it could be retained in a program-level archive. The assignment could be referenced from that archive for faculty scholarship of teaching (SoTL), as well as for program or accreditation review and other administrative purposes.
Once the assignment has been registered, the student could start from a student dashboard to request a review or feedback and guidance. We are differentiating the idea of a full review (with a rubric) from more informal feedback and guidance. This informal feedback would probably not be fed into a grade book but the captured feedback could be used by a student as evidence in a learning portfolio.
Student Dashboard (click image to enlarge)
The basic workflow for a student would let the student request a rubric-based review for a specific assignment in a specific course. The student would select the course, assignment, and other metadata. Once posted for review, the request would either be routed to multiple reviewers or the student would embed the review into a webpage using the HTML code provided. In the second step there would be an opportunity to upload a document. This might be used in cases where the document had no web incarnation (to give it a URL) or to “turn-in” a copy of the document that would not be subject to further editing, as might be required in some high-stakes assessments.
The Learning 2.0 model is supported in the last step, where the assessment is embedded in a web space still open to modification by the learner (as the previous examples illustrated)
Student-created Rubric-based Survey (click image to enlarge)
Students might want to use their own rubric-based surveys. This mock-up shows how workflow would branch from the previous workflow to allow the student to define rubric dimensions and criteria.
Student-created Simple Feedback Survey (click image to enlarge)
This last example shows how the student would create a simple feedback survey.
State of the Art
Presently there are several tools that might be used to deliver a rubric survey. The challenge is the amount of handwork implied to let each student have a rubric survey for each assignment in each course and to aggregate the data from those surveys by assignment, by course, and by student for reporting. A future post will explore what might be learned by having the data centrally aggregated. If there is value in central aggregation, it will have implications for the tool and method selected for the rubric survey delivery. The Center for Teaching Learning and Technology at WSU already has tools to make the handwork implied tractable for use in a pilot course of 20-30 students. We understand the path to develop further automation, but both pilot test and further automation require investment which requires further analysis of commitment, infrastructure, and resources.
Questions
1. Can this concept provide transformative assessment data that can be used by students, instructors, and programs to advance learning? In addition to assessing student learning, can it provide data for instructor course evaluation and for program level assessment and accreditation?
2. Can the process be made simple enough to be non-obtrusive in terms of overhead in a course’s operations?
3. Is it necessary to implement a feedback and guidance as a central university-hosted tool, or could students implement an adequate solution without more investment on the university’s part?
Previously we (Theron DesRosier, Jayme Jacobson, and I) wrote about implementing our ideas for a transformed grade book. The next step in that discussion was to think about how to render the data. Our question was, “How do we render this data so that the learner can learn from it?”
We’ve been spending time with Grant Wiggins’ Assessing Student Performance, Jossey-Bass, 1993. In Chapter 6 on Feedback, he differentiates ‘feedback’ from ‘guidance’ in several examples and defines feedback as “information that provides the performer with direct, usable insights into current performance, based on tangible differences between current performance and hoped-for performance.”
Wiggins describes ‘guidance’ as looking forward, the roadmap to follow to my goal and ‘feedback’ as looking backward, did my last action keep me on the road or steer me off?
Wiggins points to Peter Elbow, Embracing Contraries and Explorations in Learning and Teaching, “The unspoken premise that permeates much of education is that every performance must be measured and that the most important response to a performance is to measure it.” Both authors go on to suggest that less measurement is needed, and that feedback is the important (often missing) element to substitute for measurement.
Our previous posts on the transformed grade book are designed to be a measurement strategy (that we hoped would also provide feedback to learners), but Wiggins leads me to think that learners need a feedback tool different from a measurement tool (used to populate the grade book).
While bloggers have some experience getting feedback from the blogosphere by informal means, I think that it would be useful to scaffold requesting feedback, for both learners and feedback givers. However, I want the process to be simple and fast for both parties. What I hope to avoid is the too common tendency of student peers to give trite “great job” reviews, or to fall into reviewing mechanical things, such as looking for spelling errors.
To that end, I am exploring a simplified approach from the idea in the last post. Recently, I tried a version of this simple idea in an email to Gary Brown (our director). He had asked me for a report on LMS costs to be shared with a campus committee. I replied with the report and this:
Feedback request: Is this going to meet your needs for the LMS committee? (Yes/somewhat/no)Guidance request: what additional issues would you like considered?
Implicit in the feedback request was my goal of meeting his needs.
Even with this simple feedback + guidance request the question remains, can we render the data that would be collected in a way the learner could learn from it? Below is a hypothetical graph of multiple events (draft and final documents) where I asked Gary for feedback: “Is this useful?” The series makes evident to me (the learner) that initially the feedback I’m getting is not very affirming, and final versions don’t fair much better than drafts. Reflecting on this I have a heart-to-heart talk and afterwards develop a new strategy that improves my feedback results.
Versions of this kind of “Was this helpful?” feedback appear on some online help resources, and I assume that someone is reviewing the feedback and updating the help pages, and could produce graphs similar to the one above, showing improved feedback after specific interventions.
Here is Google’s feedback request from a help page found from a Google App:
When you choose the Yes or No feedback, another question appears, and in this case you are giving guidance on what would make the item better, either picking a guidance from a list or providing an open-ended reply.
In addition to comments or trackbacks, please give me feedback and guidance on this post (my form is not as slick as Google’s).
Loading…
Nils Peterson, Theron Desrosier, Jayme Jacobson
CTLT has been thinking about portfolios for learning and their relationship to institutionally supported learning tools and course designs. This thinking has us moving away from the traditional LMS. It has also led to a recognition that grade books are QWERTY artifacts of Learning 1.0. In a recent Campus Technology interview Gary Brown introduced the term “harvesting gradebook” to describe the grade book that a faculty needs to work in these decentralized environments.
“Right now at WSU, one of the things we’re developing in collaboration with Microsoft is a ‘harvesting’ gradebook. So as an instructor in an environment like this, my gradebook for you as a student has links to all the different things that are required of you in order for me to credit you for completing the work in my class. But you may have worked up one of the assignments in Flickr, another in Google Groups, another in Picasa, and another in a wiki.”
This post will provide more definition and a potential implementation for this new kind of transformed grade book. It is the result of a conversation between Nils Peterson, Theron DesRosier and Jayme Jacobson diagrammed here.
Figure 1: White board used for drafting these ideas. Black ink is “traditional” model, Blue is a first variation, Red is a second variation.
The process begins with a set of criteria that is agreed by to be useful by a community and is adopted across an academic program. An example is WSU’s Critical Thinking Rubric. This rubric was developed by the processes of a “traditional” academic community. How the process changes as the community changes will be discussed below.
Instructors start the process by defining assignments for their classes and “registering” them with the program. Various metadata are associated with the assignment in the registration process. Registration is important because in the end the process we propose will be able to link up student work, assessment of the work, the assignment that prompted the work, and assessments of the assignment. More implications of this registration will be seen below.
The student works the assignment and produces a solution in any number of media and venues, which might include the student’s ePortfolio (we define ePortfolio broadly). The student combines their work with the program’s rubric (in a survey format). The rubric survey is administered to either a specifically selected list of reviewers or to an ad hoc group. (We have been experimenting with two mechanisms for doing this “combining.” One places the rubric survey on the page with the student’s work as a sidebar or footer (analogous to a Comment feature, or the “Was this helpful?” survey included in some online resources). This approach is public to anyone who can access the web page. The other strategy imbeds a link to the student’s work in a survey, it can be targeted to a specific reviewer. This example comes from the judging CTLT’s 2nd ePortfolio contest.
In either case the survey collects a score and qualitative feedback for the student’s work. We are imagining the survey engine is centrally hosted so that all the data is compiled into a single location and therefore is accessible to the academic program. Data can be organized by student, assignment, academic term, or course. A tool we are developing that can do this is called Skylight Matrix Survey System, which is rebranded as Flashlight Online 2.0 by the TLT Group. The important properties of Skylight for this application are the ability to render a rubric question type and the ability to have many survey instances (respondent pools) within one survey and both report instances individually and aggregate the data across some/all the instances.
Audiences for this data
The transformative aspects of this strategy arise from the multiple audiences for the resulting data. We have labeled these collections of data, and the capacities to present the data to audiences “assessment necklaces”
Figure 2: Diagram of rubric-based assessment. Learners, peers, and faculty are shown collecting data from rubric-based assessment of portfolios, then reflecting on and presenting the multiple data points (necklaces) in contexts important to them.
Students can review the data for self-reflection and can use the data as evidence in a learning portfolio. We are exploring ideas like Google’s Motion Chart gadget (aka Trednalyzer/Gapminder) to help visualize this data over time. They can also learn from giving rubric-based reviews to peers and by comparing themselves to aggregates of peer data.
Instructors can use the data (probably presented in the student’s course portfolio) for “grading” in a course. It’s worth noting that the Instructor’s assignments can be assessed with the same rubric, asking, “To what extent does this assignment advance each the goals of this rubric?” With the assignment rated, instructors can review the data across multiple students, assignments, and semesters for their own scholarship of teaching and learning (SoTL). Here the instructor can combine the rubric score of an assignment with the student performance on the assignment to improve the assignment. Instructors might also present this comparison data in a portfolio for more authentic teaching evaluations.
In this example the assignment might be rated by students or the instructor’s peers. Below, the rating of the assignment by wider communities will be explored.
Academic Programs can look across multiple courses and terms, for program-level learning outcomes and SoTL. They can also present the data in showcase portfolios used for recruiting students and faculty, funding and partners. This is where the collective registration of the assignment becomes important. The program can access the assignment in the context of the program, with an eye to coordinate assignments and courses to improve the coherence of the program outcomes.
The community, which might include accrediting bodies, employers and others, can use the data, as presented in portfolios by students, instructors, and the academic program, to reflect on, or give feedback to, the academic program. Over time, an important effect of this feedback should be to open dialogs that lead to changes in the rubric.
Variations on this model
The description above is still traditional in at least two important ways: the program (ie faculty) develop the rubric and the instructor decides the assignment. Variants are possible where outside interested parties participate in these activities.
First variation. WSU and University of Idaho run a joint program in Food Science. We have observed that the program enrolls a significant number of international students, from nations where food security is a pressing issue. We imagine that those nations view training food scientists as a national strategy for economic development.
We have imagined a model where the students (in conjunction with their sponsoring country), and interested NGOs, bring problem statements to the program and the program designs itself so that students are working on aspects of their problem while studying. The sponsors would also have an interest in the rubric, and students would be encouraged (required?) to maintain contacts with sponsors and NGOs and cultivate among them people to provide evaluations using the rubric.
The processes and activities described above would be similar, but the input from stakeholders would be more prominent than in the traditional university course. Review of the assignments, and decisions about the rubric, would be done within this wider community (two universities, national sponsors and NGOs). The review of assignments and the assessment of the relationship of assignments and learning products creates a very rich course evaluation, well beyond the satisfaction models presently used in traditional courses.
Second variation. This option opens the process up further and provides a model to implement Stephen Downes’ idea in Open Source Assessment. Downes says “were students given the opportunity to attempt the assessment, without the requirement that they sit through lectures or otherwise proprietary forms of learning, then they would create their own learning resources.”
In our idea of this model, the learner would come with the problem, or find a problem, and following Downes, learners would present aspects of their work to be evaluated with the program’s rubric, and the institution would credential the work based on its (and the community’s) judging of the problem/solution with the rubric. This sounds a lot like graduate education, the learner defines a problem of significance to a community and addresses that problem to the satisfaction of the community. In our proposed implementation, the ways that the community has access to the process are made more explicit.
In this variant, the decision about the rubric is an even broader community dialog and the assessment of the instructor (now mentor/learning coach) will be done by the community, both in terms of the skills demonstrated by students that the instructor mentored, and by the nature of the problems/approaches/solutions that were a result of the mentoring. The latter asks, is the instructor mentoring the student toward problems that are leading or lagging the thinking of the community?
Examples
For some sense of learning portfolios created by the processes above, consider these winners from CTLT’s 2007-08 ePortfolio contest.
The following two winners are examples of the second variant, where students were paired with a problem from a sponsor:
The Kayafungo Women’s Water Project documents the efforts of Engineers Without Borders at WSU (EWB@WSU) who partnered with the Student Movement for Real Change to provide clean water to 35,000 people in Kayafungo, Kenya.
The EEG Patient Monitoring Device portfolio follows the learning process of four MBA students who collaborated with faculty, the WSU Research Foundation, inventors, and engineers to develop a business plan for a wireless EEG patient monitoring device.
The next two are examples of the second variant — student defined problems assessed by the community. In the latter case, the student is using the work, both her activism in the community and her study-in-action as her dissertation:
The Grace Foundation started with a vision to create a non-profit organization that would assist poor and disenfranchised communities across Nigeria in four areas: Education, Health, Entrepreneurship and Advocacy. The author used the UN online volunteering program to form a team to develop a participatory model of development that addresses issues of poverty eradication from a holistic manner.
El Calaboz Portfolio chronicles the use of Internet and media strategies by the Lipan Apache Women’s Defense, a group that has grown in national and international prominence the last 75 days, from less than 10 people in August, to an e-organization of over 312 individuals currently working collectively. It now includes NGO leaders, tribal leaders, media experts, environmentalists, artists, and lawyers from the Center for Human Rights and Constitutional Law. It recently received official organization status at the UN.
The next steps in this work at WSU are to build worked examples of these software tools and to recruit faculty partners to collaborate in a small scale pilot implementation.
