Demonstrating the Importance of Equating in Performance Assessment Using Generalizability Theory

• In Event: Educational Measurement Issues in the Health Care Professions

Abstract

Generalizability (G) theory can be applied to investigate the possible sources of score variability across multiple factors (e.g., raters) of a performance assessment. In this study, we used G-theory to identify potential sources of construct-irrelevant variance using operational data from the United States Medical Licensing Examination® Step 2 Clinical Skills (CS) performance assessment (PA). We found the composition of score variance across testing locations may differ, due in part to the unique content provided at each location. Then, we demonstrated the effectiveness of using equating to adjust for sources of construct-irrelevant variance identified using G-theory. Our findings highlight the utility of G-theory and the importance of equating in PAs that seek to compare examinees equitably across factors like content differences.

Authors

• Amanda E Gorham, University of Massachusetts - Amherst

• Jerusha J. Henderek, National Board of Medical Examiners

• Amanda Clauser, National Board of Medical Examiners