Evaluating the Impacts of Multiple Writing Prompt Rater-Score Designs on Characteristics of Mixed-Format Tests

• In Event: Roundtable Session 28
  In Roundtable Session: How Reliable Are Raters?

Abstract

A natural extension of the inter-rater reliability and rater-effects research is how the calibrations of the WP scores obtained under various rating and weighting methods impact the psychometric characteristics of mixed-format tests. This study investigated the characteristics of ELA Assessments for three grades containing Common Core State Standards aligned WPs; where different types of the WPs are scored by a single rater, or two raters with scores summed, averaged, or the second score considered as a virtual item. Preliminary results for grade 4 showed that the test characteristics curves were very similar across the methods. However, summing two raters' scores exhibited lower raw score reliability as measured by Cronbach's alpha and higher standard errors in the lower ability ranges.

Authors

• Thakur B. Karkee, Measurement Incorporated

• Winnie K. Reid, Measurement Incorporated