Supporting the Valid Use and Interpretation of Student Ratings of Instruction Through Analysis, Open Dialogue, and Communication

• In Event: 51.039 - Exploring the Contested Narratives of the Value of Student Evaluations of Teaching

Abstract

Our university has student ratings of instruction (SRI) as a key focus. For over five years we have progressively implemented a cascaded course evaluation framework. This framework began with a set of common institution-wide items, and we have since progressively worked with faculties, departments, and programs, to design context-specific items. Our overarching goal is to shift the conversation about SRIs from whether they have valid uses and interpretations, to how we can ensure valid uses and interpretations. There are three key means by which we support this shift:

1) We conduct and report statistical analyses on our data
Instruments, however designed or derived, should be validated for any particular context and/or use. To this end, we have extensively analyzed the reliability and construct validity of our instrument. Of particular focus has been validation of our five-item subscale, representing institution-wide teaching and learning priorities. These analyses have both supported the validity of this measure and our items more widely, as well as facilitated further analyses that provide in-depth and essential information for interpretation. For example, we have found a consistent negative correlation between class size and our key institutional sub-scale. By communicating these results we aim to show that we do not take the instrument at face value, and demonstrate how the data can help support valid use and interpretation of SRIs.

2) We emphasize strengths and limitations of SRI’s
We relate research evidence that shows course evaluations have empirically supported strengths/uses. For instance, they can accurately report teaching strategies employed and the relative workload of a class. However, we are equally clear that course evaluations have limitations. For one, they provide a limited perspective, and therefore should not be considered exclusively, especially in summative assessments of teaching effectiveness. As well, they are not appropriate summative measures of a number of elements which sometimes appear on evaluations elsewhere. For instance, SRIs are poor indicators of students’ learning outcomes and of instructor’s domain knowledge. Importantly, our implementation process provides us with a direct opportunity to prevent the use of questions known to be problematic or biasing.

3) We actively communicate and support valid uses of SRI’s
Acknowledging that SRI validity is ultimately contingent on use and interpretation, we proactively communicate our analyses and perspective on SRIs. We accomplish this through collaborative discussions and item design during implementation, presentations, resources, and regular reporting. This includes providing guidance on the course evaluation reports themselves, where we aim to provide and explain data to our stakeholders in ways that support intended and useful interpretations of the data. For instance we provide in-line explanations on how to interpret statistics, as well as key caveats with regards to information provided. This includes drawing attention to histograms as a key element of interpretation, and cautioning readers not to make direct comparisons of SRIs between different contexts (e.g., course types, course sizes).

Together we feel that these measures have been essential in supporting the credibility and valid uses of course evaluations at our university.

Author

• Gregory Hum, University of Toronto