You are currently browsing the tag archive for the ‘survey’ tag.
In response to an earlier post regarding online course eveluation response rate considerations, Jenny Franklin noted the importance of high response rates in small courses because of small sample size. I thought it would be interesting to look at the response rate vs course size distribution. In a college where the Assistant Dean has been a proponent of online evaluations, and has worked diligently on raising response rates, I gathered the data from 348 courses (the whole college sample in Spring 2008).
The x-axis of this graph was scaled to include only courses with enrollments less than 75. That amounted to trimming off about a dozen courses with higher enrollment, including one 200+ student course. The graph is not meant to refute Jenny’s comment, but to provide a picture of the actual experience at WSU. If one were to hypothesize that this wide distribution is likely (and might be difficult to alter), what strategy might one take to course evaluations?
CTLT has been developing a new online survey tool to meet our needs to administer online course evaluations on a large scale. This work is funded in part by a FIPSE grant (BeTA) in collaboration with the TLT Group. One of the by-products we have found in the system we are calling “Survey to make a survey.”
The general idea is that one survey is used to gather information needed to create a second survey and that the data from the first survey can be moved into the second survey by simple, mostly automatic processes: the data in an Excel report coming from the first survey is transformed using Excel functions to be the data needed to create a variant of the survey for a group of respondents in the second survey.
Our first implementation of this idea is a survey exploring instructors’ (largely informal) mid-course formative assessments. The goal was to learn about good practices faculty employ, offer opportunities to learn about other practices, AND, learn which faculty would like to have a formal mid-term course evaluation administered online.
This “survey to make a survey” technique has many possible applications. In describing Matrix Surveys examples, the TLT Group gives an example in the section “Using a Survey of Faculty to Create a Matrix Survey of their Students”
An even more elaborate application of the technique would be to implement our ideas of a transformed grade book. In that example, there are several surveys that are linked to one another and that are used to report results to multiple audiences. This might be done in the semi-automatic means described above, but for uses on a large scale, the linkages enabling survey creation will require greater automation.
We have had various anecdotal evidence that incentives have limited impact on large scale course evaluations. One piece of this evidence came from a comparison of the College of Agriculture Human and Natural Resource Sciences (CAHNRS) and the College of Engineering and Architecture (CEA). Each organization surveyed ~300 course sections and ~10,000 total enrollments. CAHNRS allows instructors to offer extra credit if they choose, CEA forbids faculty geting the data that would make offering extra credit possible. CEA offers a drawing for gift certificate. They award 20 at $20 and 2 at $50 each semester. Our observation is that the the response rates in the two colleges vary for complex reasons that do not suggest extra credit is explanatory.
Additionally, CEA has reported that for Spring 2008 only 9 of the 22 awards were picked up by students, despite modifying the procedure to gather the students preferred email address in the drawing, rather than using the university issued email address. The implication is that students are not incented by this level of monetary reward.
The graph below is a comparison of response rates to an online course evaluation in a college at Washington State University. Faculty had hypothesized that the response rate was limited by the amount of time the survey was open to students so the time available was varied in two administrations:
Fall 2007 11/23 to 12/22 (29 days) 10582 possible respondents
Spring 2008 3/31 – 5/5 (35 days) 9216 possible responents
The x-axis in the graph below is time, but normalized to the % of time the survey was open.
The y axis is normalized also, to the number of responses possible, based on course enrollment data, i.e., Y is response rate.
Figure 1 (click to enlarge)
The Fall 2007 survey ran to completion (reached an asymptote) where the spring one was (maybe) still rising at the cut off date. Fall and Spring total response rates are very similar, suggesting that more time when the survey is open has little impact on total response rate. So, contrary to what faculty hypothesized, the same overall response rate was achieved in a longer surveying window. This aligns with other data we have on response rates — there is some other factor governing response rate that is not yet identified.
Its interesting to note that you can see waves in the spring data, as if faculty exhorted students on Monday and got another increment of response.
The following is an email exchange between Gary Brown, Nils Peterson of Center for Teaching Learning and Technology at WSU and members of the TLTGroup.
Ehrmann@ TLT: Nils, I’ve gotten a couple questions from a subscriber. Do any WSU colleges conduct student course evaluations exclusively online? All of them? What kind of response rate does WSU get to online surveys and what strategies seem to work best for that purpose?
Nils: WSU has several colleges that do online surveys exclusively (Engineering & Architecture; Agriculture and Natural & Human Resources; Pharmacy). Response rates vary by course from very low to 100%. Gary Brown can take up this conversation to talk about what we do/don’t know what drives response rates
Gary: As Nils notes, we have several colleges doing online evaluations, some exclusively, more joining all the time. Response rates vary, but maybe more importantly, so do the instruments and, more importantly yet, the way the evaluations are used. I won’t go into detail about the differences in the evaluations instruments we’ve encountered, but online or not, the quality and fit for a variety of pedagogies is for me much more of a concern than the mode of delivery. The way they are used extends validity, because response rates matter little if results are ignored by faculty, misunderstood or difficult to interpret, and, all too common, boiled down to a single number for ranking purposes. It is hard to make arguments about the validity of an instrument and process if it is all capped by use that is itself invalid. But that makes the more important argument—it isn’t response rate and subsequent issue of response bias that matters as much as it ought be making sure that the response is representative and appropriate for the purpose of the process—hopefully for improving students’ learning experiences.
All that aside, response rates:
In our College of Agriculture, the response rate was 53%. But that number varies widely across departments. Here is a picture of response rates across departments from about a year ago:
Needless to say, the variance across departments is mirrored by similar and dramatic variance among courses/faculty, so it is hard for us to attribute the variance exclusively to the medium of delivery. We make other conjectures in our analysis in an article we published a while back. A key to response rates, we note in the article, is that in department with the higher response rates, the chairs of the departments were involved in the design of the instrument and the decision to put it online. So there is something important to be said for leadership and the engagement in the process of that leadership. We also point to other associations with higher rates we tracked in certain classes, mostly associated with the engagement of faculty in the process, their demonstration throughout the term that they listen and respond (not necessarily capitulate) to students’ concerns, and that they work overtly to engage students in the teaching/learning/assessment process.
The issue is pretty hot, too, and there are a number of discussions about response rates:
http://www.utexas.edu/academic/diia/assessment/iar/teaching/gather/method/survey-Response.php
http://www.aapor.org/bestpractices
Most of these suggest, as you will see, that 50% is adequate, if not stellar. (The most authoritative is the last link, and they say, too, that 50% is ok.) The larger concerns I infer from your note is the utility of responses at low rates (we’ll let others worry for the moment about the implications of comparing results, as some chairs do, when the response rates differ significantly).
But our own work here at WSU with the College of Engineering suggests that the response bias may be less salient than one would presume.
We have not written this up yet, but here is a comparison of online versus paper done with the college of engineering at WSU. We have shared this with a work group from the American Evaluation Association (AEA) and are finding others who report the same phenomenon. The response rate online was about 51%, paper in class at about 71% (which is much lower than most people believe is the case for traditional paper-based, with the presumption that it runs closer to mid 90s). The samples are convenience samples based upon faculty preference for using paper or trying the online. The graph reflects 26 student evaluations randomly drawn from each of the three groups. If there is some kind of response bias, the picture here does not reveal it. We have been monitoring this as we move more and more online and remain interested in exploring the distinctions we may get (or not) when populations complete the instruments voluntarily, for extra credit, or when they are required to do so.
