A grade book traditionally is a one way reporting mechanism-it reports to students their performance as assessed by the instructor who designed the activity.   Learning  from grades in this impoverished but pervasive model is largely one way-the student learns, presumably, from the professor’s grade.  What does a student really learn from a letter or number grade? What does the faculty member learn from this transaction that will help him or her improve?  What does a program or institution learn?  We are exploring ways to do better.

There is a second goal embedded in this effort. In a Web 2.0 environment, an environment where social networking and learning are ubiquitous like the tools that are supporting that exchange, how might one, be they student, faculty, or organization, harvest that learning, deepen it, and disseminate it more broadly.
Further, and in very practical terms, students are swirling, that is, they are attending one or more institutions simultaneously and several in their educational career.  This is happening even as:

• institutions are presuming that retention is of utmost importance,
• standardized testing is being implemented so that consumers of all stripes might compare educational “products” and
• the importance of diversity is finally gaining real support.

The implications of this is deeply challenging on many levels.  How can one gather feedback from multiple sources and stakeholders at all levels to enrich our understanding about learning? We are exploring potentially more productive alternatives.

This post is in a series exploring issues related to transforming the grade book. We are seeking to apply the ideas of Transformative Assessment to grade books in order to develop a tool that will help the institution learn and improve at all levels, from students on assignments, to instructors teaching courses, to programs adjusting to meet the needs of communities (graduate schools, employers, society at large).

Much has been written lately about the Collegiate Learning Assessment (CLA), including some significant detractors (NYTimes & Chronicle of Higher Education)
Our interest here is exploring how the data that might be gained from the CLA or from our Transformed Grade book might be useful for the following purposes:

• Students analyze their own performance and growth over time
• Faculty advise students on courses, plans of study, areas for focus
• Faculty analyze the utility of their assignments in facilitating student learning
• Faculty analyze their assignments’ alignment with program goals
• Academic programs collect better profiles of students applying to major
• Academic programs analyze courses relative to program goals
• Community (employers, grad school, society) examine and discuss both the goals and the success of programs
• Demonstration of reliability in each of the assessments above

Figure 1. (click to enlarge) CLA report of results (left) and WSU Transformative Grade Book (right). Information to help read the CLA graph can be found on page 8 of this PDF. Information on reading the WSU multidimensional graph can be found here.

In the graphs above, the CLA data provides no information to the student, and no information to the instructor or program in terms of what/ how to change to alter the CLA graduation score outcome.
The CLA reduces a whole entering or exiting class of students to a single number. This is too much “chunking,” it does not inform individuals, faculty, or programs enough about the learning outcomes or the places improvements can be made to reach them.

Summary
By making the rubric and the assessment process public and using it across multiple assignments and courses, the potential to analyze and respond to the assessment is much greater. While the multidimensional graph shown compares three perspectives on one assignment, it could just as easily compare scores for a student or group recorded at different points in time.

Examples of how a transformative assessment approach can address multiple audiences and needs:

Students analyze their own performance on assignments and over time
The figure below shows a “grade book” where the grades for each assignment are recorded as a multidimensional graph. A student would see only one row, the instructor would see all rows.  In this example the later assignments are layered on top of their predecessors for comparisons.  (Data for this example are hypothetical, but were created with particular student abilities and stories in mind.)

Figure 2. Multidimensional Transformed Grade Book (click figure to enlarge)

Faculty advise students on courses, plans of study, areas for focus
In the grade book above, imagine that the last column were an average “final grade” across the preceding assignments. It seems clear that students’ different abilities could be seen and these differences used by faculty for advising.

Faculty analyze the utility of their assignments in facilitating student learning
We are proposing that faculty’s assignments, not just student work, be scored with the same rubric (asking “How well does this assignment elicit …” rather than “How well does this student’s work demonstrate …”) In the figure below the Grade book has been augmented to show how each assignment’s average rating and individual ratings from three groups: student, faculty, community. Comparison of assignment rating to student work rating offers a chance to examine how a particular assignment is (or fails to) facilitate student success in a particular dimension.

Figure 3. Using the same rubric, comparison of average student score on 5 assignments (below bar) with the average rating of faculty, student and community on each of the assignments (above bar). (Click figure to enlarge)

This figure shows the transformed grade book with data about the assignment’s ratings paired with the average rating of student work on that assignment. Assignment ratings come from students, faculty peers, and external community.

Faculty analyze their assignments to align better with program goals
Similarly, faculty could compare their assignments and student work to the program’s goals (which is presumably the source of the rubric and the source of the “competency” threshold).

Academic programs collect better profiles of students applying to major
Many professional programs have portfolios or other performance demonstrations for admission to the major and upper division coursework.  A student’s portfolio could be scored with a rubric and the resulting multidimensional graph would communicate to the student and program the student’s strengths and weaknesses. This could benefit both screening processes and could communicate to lower division courses that nature of student deficiencies in ways that those programs would know what actions to take.

Academic programs analyze courses relative to program goals
By using the same tool to assess assignments and student products within a course and by having that tool be the definition of the program’s goals, the degree of course alignment with program goals would be readily demonstrated. Further it would be possible to review a series of courses for the ways they cumulatively contribute to program goals.

Community (employers, grad school, society) examine and discuss both the goals and the success of programs
Where courses have assignments that result in a public performance by the student (portfolio, recital, internship, etc) the community can be engaged with assessing the student and with the fit between the program’s expression of its goals and the community’s (perhaps implicit) standard.

Demonstration of reliability in each of the assessments above
One of the challenges in using diverse groups of raters and qualitative schemes such as rubrics, is measuring, maintaining, and demonstrating reliability in among the raters.  We have applied the multidimensional graph to compare ratings from multiple judges in our ePortfolio contest. The agreement (or divergence) that the graphs showed correlated well with the amount of discussion and (dis-)agreement when the judges met.