Building Item Generation Environments Using Construct-Centered Design With Learning Trajectories in Mathematics

• In Event: Harnessing Psychometric Models to Develop Next-Generation, Research-Based Assessments of Rational Number Knowledge

Abstract

Objectives
This paper describes development of an interactive diagnostic assessment system (IDAS) (Confrey et al. 2011) that focuses on diagnosing and monitoring student progress defined by a learning trajectory that evolves gradually across multiple grades. The goals of the project include: 1. creating an IDAS that assesses student conceptual growth over time, 2) providing valid and reliable real-time instructional guidance to students and teachers, and 3) supporting and enhancing classroom discourse and interactions. We report on the evolution in measurement perspectives within the project. We began with the deployment of measurement models relying on developing paper and pencil assessments, applying scoring rubrics, and used IRT models to validate the LT proficiency levels in relation to predicted item difficulties (Confrey et al, in review).

The sixteen level learning trajectory is for equipartitioning, the process of creating equal groups or parts from collections or wholes. It begins with the primitive construct of fair sharing with which even young children develop competence, and describes cognitive development through the target proficiency of understanding that n objects shared among p people results in n/pths objects per person. The learning trajectory was established through research synthesis, the conduct of clinical interviews of students of ages 5-12 years, and the development of a set of paper and pencil assessment tasks, which were then administered to 5000 students across the relevant grade levels (K-8). Rubrics were developed, against which the items were independently scored to assure reliability of scoring.

From this basis, our group has developed an IDAS using networked technology that administers diagnostic assessments in real time, producing data that can be analyzed both automatically or scored by teachers. The system includes sequestered assessments, practice and intervention modes that support interactive peer-to-peer exchange, and data accumulation and reporting capabilities. The technology affords even more opportunities: diagnostic items are generated in “generative item environments” designed from clinical interviews and paper and pencil items. These environments, linked directly to specific constructs in the LT, vary items automatically (based on specification of sample spaces), directly capture data from actions on dynamic manipulatives and tools, support both open- and closed-ended responses, and capture particular patterns of behavior and misconceptions. These multi-faceted data ensure that teachers receive rich information to support valid inferences on students’ progress. Because data are being automatically captured, we can examine student performance differences between paper and pencil versions and the dynamic affordances of the IDAS system. During late summer and fall 2011, the IDAS system will be used in small-scale teaching experiments with 2nd-4th graders; student outcomes patterns will be compared to those from similar samples of students from the previous field test. Although these results will be preliminary to a larger-scale IDAS field test, they may be of interest to researchers seeking to instantiate ways to measure growth over time and provide instructional guidance to teachers, while drawing on a careful use of measurement protocols.

Authors

• Jere Confrey, North Carolina State University

• Kenny Huy Nguyen, North Carolina State University

• Drew Corley, North Carolina State University

• Alan Maloney, North Carolina State University