Evaluating the Student Ratings of Instruction at XXX University: Unpacking Reliability, Validity, Bias, and Retention

• In Event: 51.039 - Exploring the Contested Narratives of the Value of Student Evaluations of Teaching

Abstract

The student ratings of instruction (SRI) at XXX are a core component of the overall evaluation of teaching, and play a key role in both individual and collective (program, faculty, institutional) quality enhancement and assurance. SRIs are considered only one source of evidence about the quality of teaching at XXX, and all instructors and academic units are expected to complement the SRI data with evidence of student learning outcome achievement, peer feedback (observations and/or reviews of teaching and assessment materials), other external feedback (from alumni or peers outside of XXX), and reflective self-evaluation. In practice, however, anecdotally it is understood that SRIs can at times play a disproportionately important role in the evaluation of instructors on campus, which makes the internal focus on the validity, reliability and usability of the SRI data more intense.

In order to position the SRI process in the broader literature regarding student ratings of instruction, and to inform and improve the use of student ratings in our institutional evaluation processes, the focus of this evaluation is on four different aspects of the SRI data: 1). the reliability and validity of the SRI instrument; 2). The student characteristics that influence the variability of SRI responses; 3). The instructor and course characteristics that influence the variability of SRI responses; and 4). The value of SRIs as a component of our institutional analytics related to student success and retention.

These four domains inform the specific research questions for this evaluation:
1. Do the questions in the SRI instrument accurately predict the overall ratings of instruction?
2. What student characteristics (whether related to demographics or academic outcomes) predict/explain the variance in student ratings of instruction?
3. What instructor or course characteristics predict/explain the variance in student ratings of instruction?
4. Do student ratings of instruction help predict student success and retention?

This evaluation used the SRI responses from students from the previous five years, with the addition of student and instructor demographic data from our student information system, to create a multinomial regression model for each of the four research questions. The focus of this paper is to explore issues of reliability, validity, and bias in the SRI data in order to improve XXX’s institutional interpretation and use of SRI data as part of the overall framework for evaluating teaching on campus, as well as to understand how the SRI data informs our increased understanding of student retention and success.

Author

• Bradley James Wuetherick, Dalhousie University