Development and Evaluation of Diagnostic Score Reports: Process and Recommendations

• In Event: 31.045 - Supporting Teacher Action Based on Diagnostic Assessment Results

Abstract

Actionable score reports facilitate instructional decision-making, which means that contents of the report must be interpretable and useful. While the Standards for Educational and Psychological Testing (AERA et al., 2014) indicate that score report should include information on (a) what the test covers, (b) what the results mean, (c) precision of measurement, and (d) how results should be used, they do not provide guidance for developing and evaluating the extent that report meet these needs. Zenisky and Hambleton (2012) emphasized the need for stakeholder feedback in the development and evaluation of reports. They provided guidance for score report development, including a process for developing reports that involves stakeholder input, evaluation of report prototypes, and refinement of reports over time.
While this process is important for all operational assessment programs providing score reports that are intended to be actionable and useful, it is especially critical for programs providing results for diagnostic assessment systems. Diagnostic score reports summarize profiles of student mastery rather than traditional raw or scale score values summarizing overall subject-level performance, and their contents are subject to misinterpretation. It is especially critical for the development and evaluation process to include multiple opportunities for stakeholder review and feedback to ensure that reports adequately communicate information about what students know and can do in an understandable format. This presentation describes an iterative process for developing individual student score reports for use with an operational diagnostic assessment system, the Dynamic Learning Maps Alternate Assessment System. This iterative process included data collection over a five-year period including both parent and teacher focus groups and multiple rounds of refinements to the reports. The session will describe the process used to develop and evaluate diagnostic score reports, and summarize key lessons learned for other programs interested in pursuing diagnostic score reporting to meet stakeholder needs.

Authors

• Meagan Karvonen, The University of Kansas

• Amy Clark, The University of Kansas

• Russell E. Swinburne Romine, The University of Kansas

• Neal M. Kingston, The University of Kansas