A comparison of on-line and 'in-class' student feedback surveys: some unexpected results

To facilitate a cost-effective approach to Learning and Teaching (L&T) quality assurance for units or subjects offered, some universities have replaced, or are moving towards replacing, their paperbased, 'in-class' Student Feedback Surveys (SFS) with 'out-of-class' on-line surveys. Swinburne University moved to on-line surveys several years ago; in the Faculty of Engineering and Industrial Sciences (FEIS) at Swinburne, response rates in these on-line SFS surveys are very low (typically around 10% of students enrolled) and consequently some staff reasonably contest the validity of these surveys as a measure of the L&T quality in their units. This study investigates whether the 'out-ofclass' on-line SFS surveys (with their low response rates) correlate well with the equivalent 'in-class' surveys (which typically had higher response rates of around 50%). The study will provide some research-based evidence that will assist university administrators in making an informed choice on whether to continue with on-line SFS surveys or whether to explore cost-effective options for 'in-class' SFS surveys, including the use of electronic audience polling devices (clickers). Are there any significant differences between student feedback survey responses solicited by an 'outof-class' on-line survey and the responses solicited via electronic audience polling devices (clickers) from students participating in a lecture class? The preliminary phase of this study, which is reported here, analyses data obtained from voluntary and anonymous on-line and 'in-class' SFS results for 6 FEIS units of study (semester 1 of 2011) that cover the introductory, intermediate and senior levels in engineering and science fields. The 'in-class' surveys were solicited via electronic audience polling of students who agreed to participate and were attending the lecture class in the last week of semester, and the equivalent on-line results were obtained from official university surveys, where students voluntarily participated out-of-class over a 5- week period (4 weeks before the end of semester until the beginning of the examination period). This preliminary study suggests that there are some clear differences in average SFS responses between students who participated on-line (N=148) and students who participated in-class (N=325), at least for the 6 units under investigation. For the 8 questions common to both surveys, the students who participated in the on-line survey responded considerably more positively than the students who participated in the 'in-class' survey in the last week of semester. Six unit conveners volunteered to have their unit SFS (on-line and in-class) analysed for this study, and the results of the comparison were surprising. The authors of this paper expected that students who go on-line to complete SFS surveys would be a small and non-representative sample who may be highly dissatisfied with some aspect of their teaching experience, but the results suggests that the opposite is true, and that students who are motivated to do on-line SFSs often report a more positive teaching experience than those students who complete the SFS in class. The study has reported a significant difference between on-line and in-class SFS results, at least for the 6 FEIS units investigated. This difference suggests that the university needs to consider ways of increasing participation rates in their on-line student feedback surveys or alternatively needs to develop an efficient and cost-effective 'clicker-based' in-class survey instrument.