Using Think-Aloud Interviews to Refine a Scoring Rubric for a Clinically Oriented Performance Assessment

• In Event: 67.063 - Advancements and Applications in Test Validity Research and Evaluation

Abstract

Performance-based assessment is a common approach to assess the development and acquisition of practice competencies among health professions students. Judgements related to the quality of performance are typically operationalized as ratings against success criteria specified within a rubric. The extent to which the rubric is understood, interpreted and applied by assessors is critical to support valid score interpretations and their subsequent use. Therefore, the purpose of this study is to evaluate the scoring rubric and associated performance criteria with assessors using the think-aloud method. Preliminary findings support refinement of the scoring rubric and enhanced rater training. Additionally, findings from this study support the scoring inference in a validity argument and provides insight into assessors’ cognition during the rating task.

Authors

• Mary Roduta Roberts, University of Alberta

• Chad M. Gotch, Washington State University

• Karin Werther, University of Alberta

• Kimberly Coutts, University of Alberta

• Megan Cook, University of Alberta