Using Assessment Engineering and Item Complexity Modeling to Enhance Score Interpretation

Session Type: Coordinated Paper Session

Abstract

Subject matter experts (SMEs) used assessment engineering techniques to define complexity design layers (CDLs) and develop complexity scoring protocols (CSPs) for MAP Growth math and reading assessments. Psychometricians used the ratings and derived complexity covariates (DCCs) to predict item difficulty. Purposes of the work include the development of resources to facilitate score interpretation and automatic item generation.

Math and reading SMEs followed a similar process to define the CDLs and CSPs. They drew from existing literature and content standards to define the initial versions. SMEs then went through two rounds of rating a sample of 50 items and revising the protocols. After a final round of revisions, SMEs rated a sample of 100 items in each subject.

Psychometricians analyzed the ratings. R-squared was 0.79 for the math analysis and it was 0.69 for the reading analysis. The analysis explored a variety of machine learning methods for producing DCCs and predicting item difficulty.

Two papers in this coordinated session describe work by the SMEs to define and refine the CDLs and CSPs in math and reading. The other two papers describe item difficulty modeling results using regression and a variety of machine learning methods.

Sub Unit

• Individual Papers, Innovation Demonstrations, Coordinated Sessions, or Organized Discussions

Session Organizer

• Patrick Meyer, NWEA

Chair

• Richard Melvin Luecht, University of North Carolina at Greensboro

Individual Presentations

• Developing Complexity Design Layers and Scoring Protocols for Math Items - Sarah Whitney, NWEA; Alane Ferland, NWEA; Mark Lawler, NWEA; Desiree Spikings, NWEA; Mary Resanovich, NWEA

• Math Item Difficulty Modeling Using Complexity Design Layers and Item Metadata - Xueming Li, NWEA

• Defining Text Complexity, Item Complexity, and the Text-by-Item Interaction for Reading Assessment - Gina Wilmurth, NWEA; Julie Richardson, NWEA; Adam Withycombe, NWEA; Janice Lee Johnson, NWEA

• Reading Item Difficulty Modeling with Various Machine Learning Methods - Ann Hu, NWEA; Patrick Meyer, NWEA

Discussant

• Steve Ferrara, Cognia/Steve Ferrara