Applying Kane’s Validity Framework to Student Evaluations of Teaching: Conceptual Framework and Empirical Analysis (Poster 26)

• In Event: AERA Poster Session Wednesday 10:50 a.m.
  In Poster Session: Measurement, Psychometrics, and Assessment: Poster Session I

Abstract

The validity of student ratings of instruction (SETs) for high-stakes decision-making remains highly contentious. Kane’s argument-based validity framework (Kane, 2013) can help explicate the intended interpretation and use of SETs, design validation studies, and integrate validity evidence. This study investigates the inter-rater reliability, test-retest reliability, internal consistency, and the internal structure of an SET instrument using generalizability theory and multilevel factor analysis. The data comprises 967,817 SET surveys from 131,010 students across 56,846 courses from 2018/19 to 2022/23. The results indicated good reliability, a unidimensional factor structure, and strong item psychometric properties. These findings are integrated into a coherent validity argument and contextualized within a broader validity argument. Implications for using Kane’s framework for SETs will be discussed.

Authors

• Christine M. Barron, University of Toronto - OISE

• Yuxin Tu, University of Toronto