Exploring the Impact of Score Resolution on Person Fit and Decision Consistency in Rater-Mediated Assessments

• In Event: 66.077 - Methodological Considerations of Rasch Models

Abstract

Scoring practices in many rater-mediated performance assessments include score resolution procedures, whereby a third rater adjudicates discrepancies between two raters’ ratings of the same performance. There are numerous approaches for calculating resolved scores that involve different combinations of the original and third ratings. Using data from two content-area writing assessments, we examined the impact of five resolution procedures on the values and psychometric quality of student achievement estimates. We observed substantial differences between the values and quality of student achievement estimates across methods. Our results highlight the importance of examining person fit as part of evaluating the psychometric quality of rater-mediated performance assessments in which score resolution procedures are employed.

Authors

• Stefanie A. Wind, The University of Alabama - Tuscaloosa

• Angela Adrienne Walker, Gwinnett County Public Schools

• Cheng Hua, The University of Alabama

• Abdullah Asilkalkan, The University of Alabama