Estimating the Reliability of Performance Scores: Important Considerations

• In Event: Rating and Scoring Issues in Assessment

Abstract

Though performance-based assessment provides a mechanism for assessing complex skills like clinical competence, it also poses unique challenges to the estimation of score reliability. This study uses generalizability theory to illustrate the importance of accounting for multiple sources of error, the natural structure of data, and the decision-making standard in establishing score reliability. Performance data from 152 students who completed an OSCE in 2008 were used to conduct a series of generalizability studies that took into consideration different potential error sources and study designs. Results indicate that careful consideration of multiple error sources, the naturally occurring structure of the performance data, and the standard used are needed to avoid misinterpretation of score generalizability.

Author

• Regina Richter, University of California - Los Angeles