You are currently browsing the monthly archive for September 2008.
CTLT has been developing a new online survey tool to meet our needs to administer online course evaluations on a large scale. This work is funded in part by a FIPSE grant (BeTA) in collaboration with the TLT Group. One of the by-products we have found in the system we are calling “Survey to make a survey.”
The general idea is that one survey is used to gather information needed to create a second survey and that the data from the first survey can be moved into the second survey by simple, mostly automatic processes: the data in an Excel report coming from the first survey is transformed using Excel functions to be the data needed to create a variant of the survey for a group of respondents in the second survey.
Our first implementation of this idea is a survey exploring instructors’ (largely informal) mid-course formative assessments. The goal was to learn about good practices faculty employ, offer opportunities to learn about other practices, AND, learn which faculty would like to have a formal mid-term course evaluation administered online.
This “survey to make a survey” technique has many possible applications. In describing Matrix Surveys examples, the TLT Group gives an example in the section “Using a Survey of Faculty to Create a Matrix Survey of their Students”
An even more elaborate application of the technique would be to implement our ideas of a transformed grade book. In that example, there are several surveys that are linked to one another and that are used to report results to multiple audiences. This might be done in the semi-automatic means described above, but for uses on a large scale, the linkages enabling survey creation will require greater automation.
The September 5th “WSU Today” shared the 2008-2013 Washington State University strategic plan. The four goals are commendable, challenging, and we believe attainable—achieving a national and international preeminence in innovation, discovery, and creativity; providing a premier education and transformative experience that prepares students to excel in a global society; leading in relevant local, national, and global outreach and engagement, and embracing an environment of diversity, integrity, and transparency. Providing students with a transformative experience (goal 2), it strikes us in the WSU Center for Teaching, Learning, & Technology, is central for achieving the other three goals, and the key to that goal will be identifying the benchmarks and measures used to determine if and when we have done so. If you want more of something, the saying goes, measure it. The particular challenge for measuring transformation is that old measures may be useful for assessing gains or doing more of something or doing something better, but doing something better is not the same as transforming it. Progress is not necessarily transformation. We might ultimately find more oil to burn or even make engines more efficient at burning it, but it is still burning oil. To bring the point back to transforming teaching and learning, it may be useful to recognize that most students who graduate tomorrow will be doing jobs that have not yet been invented, just as most successful businesses and business ventures will be producing something different or doing something very differently if they are to succeed and achieve preeminence. Measures that demonstrate WSU is doing something as well or better than our peers as those proffered in the strategic plan, measures like recruiting higher SATs or increasing enrollments and graduating more of this or that major, will not fully reflect the kinds of transformed learners WSU prepares and graduates. Engaging WSU students more fully or providing them with better support in business as usual does not and will not by itself illustrate the kinds of transformative experiences many of WSU students already receive (though engaging more students in more of those experiences are certainly two goals to strive for).
At WSU’s Center for Teaching, Learning, & Technology, we work with hundreds of WSU faculty every year to meet the transformative challenge. We help them integrate new technologies like wikis, blogs, and ePortfolios not just for the sake of meeting students’ every increasing technology proclivities, but to apply those tools and the knowledge they are gaining to solve real problems and meet new ever mutating challenges. For examples of transformation as well as transformative ways of measuring that change, we invite you to peruse the CTLT Blog and website for examples of transformative learning already transpiring at WSU.
George Siemens interviewed Dave Cormier on his recent article in Innovate, Rhizomic Education.
I found this 3 minute portion of the interview more lucid than the article itself regarding community, curriculum and learning in context. It argues, “people will learn by being part of a community (that has a purpose).”
In the interview, Cormier speaks of the course he taught summer 08 – Educational Technology and the Adult Learner. He describes his approach to the course as crediting the student as having their own ability to create knowledge. His premise is that all that is to be known is out there and the learner’s task is to contextualize what is known to the learner and the learner’s needs.
Cormier describes the product of the course as a “textbook” written by the students, but more than an object, he calls it a set of skills and knowledge that students would build out in their own lives (i.e., their professional practice).
I wanted to explore the learning outcomes of the course at greater depth. In his day-8 reflection Cormier says: “What we did not focus on was outlining the ‘takeaways’ that students needed to bring out of the course itself, at least, not in a communal sense. … In a very real sense, each of those students will be taking a very different set of takeaways from this course, related to what they themselves put in, how they contributed to the community and where they are going to take those new literacies when they go back to their own professional practice.”
This didn’t really satisfy me. Take aways are fine, but the students also needed to know where they sit within the community’s norms, the profession’s norms. In following the ideas in Stephen Downes’ Open Source Assessment I think Cormier should have provided the assessment used in the course for community inspection. If I understand Downes’ premise, what is missing in Cormier’s course is the assessment criteria. It is especially important in Cormier’s course because the course-as-community is developing the curriculum — the learners need to be in conversation with the wider community about norms of performance.
With the assessment criteria, you and I could look at the students’ work to see if we could “recognize” (Downes’ term) quality learning outcomes in the products of Cormier’s course. I’m not interested in Cormier ranking the students as much as answering for myself “are they competent?”, “would I hire them?”, “or where are they making progress to those goals?” Those are questions that have an authenticity to a community of practice, such as Cormier seems to be describing.
At WSU we have been thinking about community roles in shaping the assessment criteria for student work and how that changes the ways we think about the grade book. Among those ideas is that the assessment is more than a single number in a gradebook: it might be useful to express numbers along several dimensions of skill, its criteria should be open to community discussion, and the assessment should do more than measure the student, it should inform the student, assignment, course, academic program, and community (that is, it should be a transformative assessment). Consequently, we have been thinking about how the community would have access to the assessment criteria for two purposes (beyond assessing the students): 1) assess the assignment — is this assignment driving at producing the competencies that it holds the students to and, broader, 2) is this assessment measuring the competencies that the community values. Its in this latter sense that Downes’ Open Source Assessment piece is important to our thinking – beyond community as curriculum to community as assessment.
Tag Cloud of contest winners
The HASTAC and MacArthur Foundation Digital Media and Learning Competition results are posted. I found several interesting projects, but what I found most interesting was the tag cloud made from the winners’ project descriptions. The words have me thinking about some of the ideas we surfaced in our ePortfolios case studies. In one perspective collaboration is cheating and technologies are needed to prevent it. Another perspective is that collaboration (we is smarter than me) is the inherent message of the Internet.
We have had various anecdotal evidence that incentives have limited impact on large scale course evaluations. One piece of this evidence came from a comparison of the College of Agriculture Human and Natural Resource Sciences (CAHNRS) and the College of Engineering and Architecture (CEA). Each organization surveyed ~300 course sections and ~10,000 total enrollments. CAHNRS allows instructors to offer extra credit if they choose, CEA forbids faculty geting the data that would make offering extra credit possible. CEA offers a drawing for gift certificate. They award 20 at $20 and 2 at $50 each semester. Our observation is that the the response rates in the two colleges vary for complex reasons that do not suggest extra credit is explanatory.
Additionally, CEA has reported that for Spring 2008 only 9 of the 22 awards were picked up by students, despite modifying the procedure to gather the students preferred email address in the drawing, rather than using the university issued email address. The implication is that students are not incented by this level of monetary reward.
