From Rasch Models to Rule Space and Poset-Based Adaptive Testing

• In Event: New Measurement Paradigms: Psychometric Methods for Technology-Based Assessments

Abstract

We present an illustrative case of the use of Item Response Theory (IRT) to develop and evaluate an assessment based on learning trajectories and the move to more sophisticated theories and technologies. The published version of our assessment measures core mathematical abilities of children from age 3 to 9 years using an individual interview format. Abilities are assessed according to theoretically- and empirically-based developmental progressions that underlie research-based learning trajectories that we consider central for diagnostic assessment.
We defined mathematical competence as a latent trait in IRT, yielding a score that locates children on a common ability scale with a consistent metric. All items are ordered by Rasch item difficulty; children stop the assessment after four consecutive errors. Previous analysis of the assessment data showed that its reliability ranged from 0.75 to 0.94 on the subtests and 0.93 to 0.94 on the total test scores (Clements, Sarama, & Liu, 2008). Despite advances, the assessment has not met the additional criteria established by the field, including the following.
1. Be administered in a reasonable period of time.
2. Avoid frustrating children with low levels of achievement, or wasting time in assessing children with high levels of achievement.
3. Yield information about each child’s concepts and skills in each of multiple domains or topics of mathematics.
4. Provide information about specific concepts and skills that constitute the cognitive components of thinking involved in various tasks.
5. Provide information about general processes involved in successful mathematical thinking across several items and topics.
6. In so doing, provide profiles of individual children that are more informative than a single score and immediately useful for formative assessment.
7. Allow a fine-grained evaluation of the theorized developmental progressions and their inter-relationships, thus informing future research, curriculum development, and pedagogical efforts.
To address these shortcomings, we are creating a reduced measure, the Comprehensive Research-based Early Math Ability Test (CREMAT) that will use Computer Adaptive Testing (CAT) to guide the dynamic selection of items and score and analyze the results, providing a comprehensive report including the overall score, level of achievement within each developmental progression (and, if requested, detailed report on achievement on each item in that topic), and a cognitive profile of attributes. We are conducting cognitive and statistical analyses of the an existing research-based mathematics assessment using the theories and procedures of Kikumi and Curtis Tatsuoka, including Q-Matrix theory, the Rule Space Method, and poset-based adaptive testing methodologies. We will produce a reduced and adaptive assessment that will take less than one-fourth to one-half the time to administer and yet will yield more useful and detailed information about children’s knowledge of mathematics, including their level of thinking along multiple empirically-validated developmental progressions and detailed cognitive profiles. In accomplishing this, we will evaluate, refine, and elaborate these cognitive developmental progressions; operationally define the cognitive attributes (e.g., concepts and skills) that constitute each levels of thinking in those progressions; and empirically evaluate the theoretical model that includes these attributes.

Authors

• Douglas H. Clements, University of Denver

• Curtis Tatsuoka, Case Western Reserve University

• Kikum Tatsuoka, Teachers College, Columbia University