Validity Evidence for Rater-Mediated Performance Assessments

• In Event: Practical Guidance and Best Practices for Gathering Validity Evidence Based on Assessment Type

Abstract

The next paper will consider validity evidence in the context of rater-mediated performance assessments. Validity considerations for these assessments are unique because all of the information about examinee achievement is directly tied to rater judgments. Accordingly, the quality of rater judgments directly impacts the validity of the interpretation and use of student achievement estimates in these assessments.

To facilitate the discussion, data will be used from a large-scale content area writing assessment that was recently administered in a large school district in the United States. The example performance assessment is a high school social studies performance assessment (N = 10,833). Students were given content-specific stimuli (e.g., informational texts, charts, tables, or diagrams) and were expected to incorporate the information from these stimuli into their responses. Two randomly selected raters from a group of 62 trained raters scored each composition on four domains: depth of development of ideas, knowledge of content, organization of the response, and writing conventions. The raters used a five-point rating scale (1 = lowest, 5 = highest).

In the operational administration of the assessment, rater agreement indicators were used to evaluate the psychometric quality of the assessment. In the proposed study, indicators based on the partial credit (PC) formulation (Masters, 1982) of the Many-Facet Rasch (MFR) model (Linacre, 1989) will be used to examine the measurement properties of the assessment because this framework offers useful indices related to rater invariance and model-data fit indices for individual students, raters, and other facets that have relatively straightforward interpretations.

Several authors have proposed frameworks for gathering and evaluating validity evidence in the context of rater-mediated assessments (e.g., Engelhard & Wind, 2018). The current study will build upon that previous work and complement the other studies in the symposium by highlighting the unique validity considerations and practical issues associated with rater-mediated performance assessments that are not relevant in other assessment formats. For example, we consider issues related to rater disagreements and score adjudication, and the impacts of using particular indices (e.g., rater agreement vs. rater fit to a measurement model) on measurement properties (i.e., person fit).

Author

• Stefanie A. Wind, The University of Alabama - Tuscaloosa