Examining the Reliability of Scores From a Performance-Based Assessment of Student Competencies

• In Event: 72.025 - Psychometric Issues and Performance Over Time

Abstract

The purpose of this study is to examine the reliability of scores from a performance-based assessment of practice competencies for occupational therapy students. Data from 99 occupational therapy students who participated in a practical exam were examined. The rater approach was used in a generalizability analysis of analytic and holistic scores to account for the nested design. An examination of the results demonstrates that on average, rater pairs produced more reliable holistic scores (i.e., G = 0.87, ϕ = 0.83) vs. analytic scores (i.e., G = 0.77, ϕ = 0.72). Findings from this study provide insight into possible differences in rater behavior raising additional questions for future research around the processes raters use to generate holistic vs. analytic scores.

Authors

• Mary Roduta Roberts, University of Alberta

• Karin Werther, University of Alberta