Exploring Assessor Cognition as a Source of Score Variability in Performance Assessment of Practice-Based Competencies

• In Event: 64.044 - Measuring Learner Differences and Variability

Abstract

The purpose of this study was to explore assessor cognition as a source of score variability in performance assessment of practice-based competencies. A mixed-method sequential explanatory study design was used where findings from the qualitative strand assisted in interpretation of results from the quantitative strand. Results from two generalizability analyses indicated the highest-order interaction-error term accounted for the second-highest proportion of variance, after student variation. Score reliability was higher in the holistic vs. analytic scoring framework. Think-aloud interviews were conducted with eight participants assessing a subset of student performances from the quantitative study. Verbal analysis of assessors’ cognitions provided evidential support for the quantitative results. Findings of this study are interpretable from the measurement and socio-cultural perspectives on assessor cognition.

Authors

• Mary Roduta Roberts, University of Alberta

• Megan Cook, University of Alberta