Getting Started With Learning Outcomes Assessment, Activities Not Abstract Outcomes

You probably saw it in the Chronicle of Higher Education (October 27, 2009) in the comments of George Kuh, director of the National Institute for Learning Outcomes Assessment.  He makes several observations that align with our experience, reading, and thinking.  He notes, for instance, that “what we want is for assessment to become a public, shared responsibility, so there should be departmental leadership” (paragraph 14).

He also notes that lots of places have developed outcomes, however:

“What’s still disconcerting is that I don’t see a lot of evidence of closing the loop. There’s a lot of data around, there’s some evidence it’s being used in a variety of ways, but we still don’t know if that information is being transferred in such a way as to change practices for the better. That’s still the place where we’re falling short” (paragraph 6).

http://chronicle.com/article/An-Expert-Surveys-the/48945/?key=HWsgcl03ZSJNZHs2K3UTKScFaHt6JkJ4Y34WMHYabFlW

Part of the reason closing the loop is so difficult is that outcomes assessment remains removed from what faculty do in their classrooms.  (There’s a nice piece in Inside Higher Ed today on this, but this post is already too long.) So what we’ve learned that tends to work better and is generally most practical is to put the focus on what faculty are already doing.

Peter Ewell, the VP of The National Center for Higher Education Management Systems, came to a similar conclusion that resonates with our experience and suggests a strategy for making outcomes assessment truly “practical” or “functional” for closing the loop.  Lamenting the failure of more than 10 years of the assessment reform in helping institutions and faculty close the loop, Ewell says:

“I have learned two additional lessons about the slippery matter of articulating abilities.  First, it’s more useful to start with the actual practice of the ability than with the stated outcome.  Phrases like ‘intellectual agility’ have great charm, but mean little in the absence of an actual student performance that might demonstrate them.  To construct assessment techniques, formal assessment design, as described in the textbooks, demands ever more detailed verbal specifications of the outcomes or competencies to be developed.  But it is often more helpful to go the other way. Given a broad general descriptor like ‘intellectual agility,’ can you imagine a very concrete situation in which somebody might display this ability, and how they might actually behave?  Better still, can you quickly specify the parameters of an assignment or problem that might demand a particular level of ability for access?  The performance that the student exhibits on the assessment is the definition of the ability itself; the ability has no independent existence” (pp. 6, 2004, General Education and the Assessment Reform Agenda).

We’ve worked with a couple of dozen programs here at WSU (and more than a few elsewhere) and found that starting with the real and embedded assignments faculty use is an effective way to approach outcomes assessment.  It helps programs refine and make concrete their understanding of outcomes in the context of their own teaching.  It helps them close the loop as reflected in their own assignment design.

Given the ever increasing pressure to demonstrate outcomes educators all face, and in the interests on building outcomes assessment systems in the day-to-day work in which faculty are already engaged, starting with activities faculty already use is what will give us the best, well, outcome;-)

2009 WSU OUE Innovation and Leadership Awards (Intro)

We live in interesting times. Like the world and the country, higher education is experiencing a confluence of forces for change. There is the explosion of technology that is changing not just how we work, but how we relate to each other. There is a renewed press for greater accountability. And there is an urgency that we work with ever greater efficiencies and in sustainable ways. Fortunately, here at WSU we have innovators and leaders who are helping us navigate the forces of change, pioneering models and methods that are helping us maintain what matters most and imagine and create what is most needed. This spring we had the opportunity to recognize some of our innovators and leaders at the Office of Undergraduate Education Award Ceremony. Each of CTLT’s Innovation and Leadership awards was presented to a colleague who has provided a new way to think about the roles and responsibilities of educators for building communities of practice, engaging authentic and pressing challenges, and doing so in ways that demonstrate a commitment to high standards of accountability. Here they are with short descriptions of the teaching and learning stories.

Faculty Teaching, Learning, and Assessment Innovation Awards – Chida

Meriem Chida arrived in the CTLT offices a week after Fall’08 semester had begun, a new hire with an idea to bring her problem-based teaching methods to Pullman. Chida had connections with experts in her industry, Fashion Forecasting, and had used a design project with expert feedback in her previous teaching.

Pullman was remote from her experts, so she was forced to pilot her methods online, and have students create “posters” that could be shared with one another and experts in portfolios (blogs). While the technologies have changed, the outlines of these ideas go back a decade.

With CTLT, Chida piloted the Harvesting Gradebook , a technique to gather structures and open-ended feedback from her three audiences: industry, faculty in the department and student peers. In addition to gathering feedback about the student projects, Chida’s “gradebook” gathered feedback on the assessment tools (rubrics) she used. This allowed the community to answer — is this way of assessing how students talk about data useful? And her work helped explore the different character of feedback provided by industry, faculty and student peers. Chida is receiving one of the Faculty Teaching, Learning, and Assessment Innovation Awards for her Scholarship of Teaching and Learning. She is exploring the development of community-based learning, where the community has a role in the discussion about what is important to learn and how it might be best assessed.

Faculty Teaching, Learning, and Assessment Innovation Awards – Tamez

When Theron DesRosier met Margo Tamez in the fall of 2006, she was under a heavy weight; an agent from Department of Homeland Security had recently told her mother, Eloisa Garcia Tamez, that her land would be “taken either by signed waiver or by force”, she had 30 days to decide. Eloisa lives on El Calaboz Rancheria, in the Lower Rio Grande Valley on the Texas-Mexico border. These are ancestral lands that were passed down to Eloisa by her Apache foremothers /forefathers. Originally, 22,000 acres were designated for the family but over hundreds of years the land has been snatched bit by bit until now only a few acres remain.

Margo felt isolated from the struggles of her community and detached from her research. She began looking for a way to unite the two. She had no financial resources but she was determined to find a way to help her mother. In desperation she decided to go onto the Internet and share her struggle with the world. She wrote a post entitled URGENT CALL!, before thanksgiving, on a prominent blog and included her cell phone number to contact. As she was traveling home for thanksgiving people from all over the continent began calling. Her voice mail filled up. People told her stories, gave advice and encouragement. She made a point to answer every voice message.

Soon after this experience Margo wrote, “I was getting the sense that my journey home, to El Calaboz, and my ‘research’ journey were joining…suddenly keeping the two in separate spheres wasn’t appropriate anymore,”

The initial success encouraged Margo to develop the strategy further. She describes it as the Web 2.0 version of a basket communication strategy her foremothers used: designs on the baskets they carried communicated important information to the community as they worked. The key elements of this strategy are transparency and utility.

She wrote in Wikipedia and Native Wiki. She nurtured, organized and mobilized a dispersed community using blogs, myspace, facebook, and text msging. She asked members of her group to document the struggle with video cameras, then upload the videos to youtube so they could be broadcast to the world. She calls these tools her “palettes, paints, glitter”

Over the next six months her organization grew to over 300 individuals including NGO leaders, legal experts, tribal Elders, media professionals, environmentalists, artists, activists, policy makers, scholars, and Native American and Indigenous organizations. They have set precedent in two landmark federal and international legal cases and have received official organization status at the UN. And her mother still lives on her homeland.

The ePortfolio that she continues to build chronicles this journey. This experience has dramatically changed the way Margo thinks about teaching, learning, and research. Her course space is now a world accessible blog that acts as a hub for student blogs. Margo’s work has been the focus of many hours of reflection at CTLT. Our understanding of eportfolios, social networking, and distributed learning has been greatly enrich by her example.  Margo’s accomplishment is also encouraging in this time of limited financial resources. With tools free to anyone on this campus, she has created a vibrant global network that joins community, research and action.

“CTLT is a core lab for my writing, thinking and being process as a researcher.”
Margo Tamez

Leadership in Teaching, Learning, and Assessment Award – Olsen

Since 2006, Dr. Robert G. Olsen has been working with CTLT on creating a college-wide meaningful assessment cycle that serves as both a catalyst and a road map for continual improvement at the individual and program levels. His leadership in this effort has been crucial – without his on-going support the college would not have made such great strides in such a short time.  The work that he has lead on the development and direct assessment of engineering professional skills has received important recognition from ABET the national accrediting agency for engineering programs and the American Society of Engineering Education.  Thank you, Bob, for your leadership. We are excited to continue our work together on this and other teaching, learning and assessment projects for optimal impact.

Faculty Teaching, Learning, and Assessment Innovation Awards – Pedrow

Dr. Patrick Pedrow has been working closely with the Center for Teaching, Learning and Technology over the last two years to re-vision the Electrical and Computer Engineering senior design sequence to integrate the development and direct assessment of key engineering professional skills, all of which can be considered global competencies and which are critical to preparing our graduates for success in the interdisciplinary, multicultural interactions that characterize 21st century engineering careers. In addition, his new course is offered as an electronic portfolio, with nested team and individual student portfolios. CTLT applauds his enthusiasm, commitment to collaboration and risk-taking that comes with trying out the new.

Shifting Faculty Roles for New Learning Environments

AAC&U Conference , April 2-4 2009

Round Table presentation

Nils S. Peterson, Assistant Director of The Center for Teaching, Learning, and Technology—Washington State University

Abstract: Washington State University has explored SharePoint (MOSS) as a lightweight learning management system (LMS) and SharePoint Mysites as an ePortfolio platform, supplemented with social networking tools and strategies. Integration between course spaces and portfolios has been done in a “hub and spoke” model. New strategies for facilitating and assessing learning necessitate a substantial change in faculty roles. In this session, participants will (1) explore the critical and integrative thinking skills that students will need in 21st century Web 2.0 learning/work environments and (2) use this exploration to reflect on novel assignments and faculty roles needed to (3) facilitate this learning.

The following documents were part of this discussion:

• Introduction, activity, and readings
• Current description of WSU’s work (see also this blog post from AAC&U Jan. 2009)
• Updated Learning Spectrum self-assessment
• Four strategies (different points on the Learning Spectrum)

Test drive the Harvesting Gradebook

We have been writing about an idea for a new approach to assessment of student work taking place in Web 2.0 environments. Gary Brown coined the term Harvesting Gradebook to describe this idea.

You are invited to try the Harvesting Gradebook as a member of the world community. This time-sensitive opportunity is part of a Washington State University course happening during Spring 2009.

Please understand that you are working with real students and an instructor in an actual running course. We appreciate your thoughtful and sensitive feedback. This blog also welcomes your meta-comments on the idea and the experience.

PROCESS

The instructor’s blog contains the assignment a series of posts which serve as the assignment prompt beginning here and adding some style guides and then some clarifications about the blog content and the assessment criteria and then a topic list. The balance of the instructor’s blog are personal reflections and modeling of activities in the assignment.

The instructor’s blog roll provides a link to the student blogs. In them you will see multiple posts by students, inviting traditional blog comments, and (in many cases) with an invitation to a “survey” that will look something like this:

Please evaluate this post:
http://skylight.wsu.edu/s/xxxxx.srv

You can preview the survey (opens in new window) (don’t post review data here, this is a just a preview ). Previewing is worth while because you will be asked to reflect on the assignment prompt, the student work and the value of the rubric itself. This 360-degree review is a reason we are calling this a transformative assessment approach.

Understand this is a work in progress, we are revising based on feedback and our own reflection. Also, understand that we are attempting to model the process by which this kind of process is evolved by the community using it, so your feedback to us (comment or trackback) is valuable. Please give us meta-comments on the process that are not appropriate for the feedback form itself.

If this experience whets your appetite to learn more about our thinking, a collection of materials can be found in a round table we prepared for AAC&U in January .

Harvesting Gradebook

CTLT has been thinking about portfolios for learning and their relationship to institutionally supported learning tools and course designs. This thinking has us moving away from the traditional LMS.  In a February 2008 Campus Technology interview, Gary Brown introduced the term “harvesting gradebook” to describe the gradebook that  faculty need to work in these decentralized environments. As originally articulated by Gary, the gradebook “harvested” student work, storing copies of the work within itself where it was assessed.

On further discussion, the concept became inverted, what was “harvested” were assessments, from work that remained in-situ.

This inversion of the idea allowed the widening of the community that could be involved in the assessment. There are ways that the instructor, as well as the program can learn from this transformed idea about a gradebook that are responsive to course and program improvements, as well as useful in accreditation.

A pilot course using these ideas earned the NUTN 2009 Best Resaerch Paper award. Here is the video made for the award ceremony.

At the AAC&U conference in Seattle, Jan 22-24, we presented these ideas at a round table on ePortfolios Friday morning. (Authentic Assessment of Learning in Global Contexts) Nils Peterson, Gary Brown, Jayme Jacobson, Theron DesRosier

In February 2009, a Campus Technology article summarized a pilot offering of a course that used this latter harvesting of assessments model beginning to demonstrate how a community could effectively participate in the process.

This post serves as a table of contents to materials from our “Authentic assessment of learning in global contexts” AAC&U presentation and background to the story in Campus Technology.

• Abstract submitted to AAC&U for the session
• The 10 point self-assessment chart handed out at AAC&U
• The booklet that was handed out (below are topics in the booklet)
  

   

  • Nature of the community feedback
  • WSU 2007-08 ePortfolio Contest (call to contestants)
  • ePortfolio Contest Results (where important lessons were learned)
    
  • Harvesting (Transformative) Gradebook
    • Assumptions behind the idea
      
    • Overview (synthesis and reflection at end of summer 08)
      
    • Inception of idea
    • Demonstration of Feasibility of Implementation (using SharePoint and Google Docs)
      
    • Radar graphs for multidimensional data (description)
    • Scaling up the Implementation (design)
    • Pilot course (Fall 08)
  • Getting the assessment embedded
  • What the gradebook looks like
  • The conversation among the community
  • Results of the pilot course (impact on learners)

Assumptions underlying the harvesting grade book

In July we made a proposal for a presentation for the AAC&U conference in January 2009 (which has been accepted, see you in Seattle).  A previous post serves as an index to the materials we are collecting for that presentation. The AAC&U call for proposals asked a series of questions that reveal their perspective on the roles of the institution, faculty, students and the wider community in the learning enterprise.

The assumptions and perspectives that appear to underlie AAC&U’s framing of the questions are summarized in the table below. The preamble to the call for proposals also expresses a concern for maintaining the relevance of colleges and universities and maintaining the importance of the values of a liberal education. We see a conflict between the AAC&U assumptions and the goal of maintaining relevance in a Web 2.0 world.

The work we are reporting at the conference comes from a different set of assumptions about the learning enterprise. The perspective is community centered. It uses a set of assumptions that have previously existed within institutions of higher learning, and to some extent, still exist at the PhD level, but which seem to have been displaced by the challenges of increasing scale and decreasing resources. We are exploring ways in which the Internet, and particularly the Web 2.0 perspective of the Internet, can facilitate a transformation to a different set of assumptions.

Central to this thinking is our work from last summer on a Harvesting (or Transformative) Grade book.  An important aspect of this re-visioning of the grade book is its move into a Web 2.0 context and its role in facilitating our assumptions about learning in communities (right column in table). This work is related to ideas Stephen Downes shared about group vs network learning and the lists below are worth examining in light of his diagram of that idea.  This little video about connectivism gives some other ideas about faculty roles and implementation of our community-centric assumptions.   Some very good comments about the role of the faculty appears ¾ of the way into the video.

Institution-centric (AAC&U) view	Community-centric View
The institution and faculty are the central node.	The learner is the central node; learners include all members of the discourse community (or Community of Practice).
The faculty member is the grading authority.	Community of Practice holds responsibility (and the social capital) for assessment, and assessment (from the root “assay” as in to sit with) is most useful when it is formative, when it is understood to be constructive feedback rather than an authority’s judgment.
Faculty define the assessment criteria.	Community of Practice, by virtue of expert consensus, validates the assessment instrument.
Feedback to the student about learning criteria is masked by the letter grade.	Students merit feedback direct and unfiltered from the community using criteria that the community articulates.
Institution is the established (credentialing) authority.	Community of Practice is the implicit credentialing authority; the university is the facilitator of that credentialing and of community building.
The unit of analysis is the course, bounded in time, content, and brick or virtual (LMS) space.	The unit of analysis is the problem, problems are not bound in time or content.
Problems for students to study are (artificially) constrained by course and disciplinary boundaries.	Communities identify authentic problems that are interdisciplinary and reach beyond the definition of the course.
Students rarely share their work with, or receive feedback from, public audiences.	ePortfolios built over multiple year are learner owned (not university owned) and used to communicate (collect, select, connect, project, and reflect) with and get feedback from wider communities.
Faculty are a gatekeeper between the students and the community of practice.	Students are anticipated to join communities of practice, faculty may introduce students to the community.
Faculty have all the social capital within both the classroom and the community of practice.	(Student) learning is social and therefore learners are building social capital within the classroom and withing the community of practice.

Related posts
1. Pilot course using the Harvesting Gradebook
2. Rich assessment data from a Harvesting Gradebook
3. Community conversation about assessment criteria
4. Differences in engaging the discussion among participant groups
5. Different conversations among participant groups
6. Evidence for Impact on Learning