Multidimensional Item Response Theory Analysis to Support Formative, Longitudinal Assessment of Early Childhood Development

• In Event: 81.028 - Supporting Children's Early Learning and Kindergarten Goals With a Standards-Aligned Assessment: Effective Practices in Statewide Aggregate Reporting

Abstract

1. Objectives
In this paper we will first explain the need for multidimensional item response modeling (MIRM) in the longitudinal assessment of early childhood development. Second, we present results from the DRDP assessment analyzed with a multidimensional Rasch model and a Delta Dimensional Alignment (DDA; Schwartz and Ayers, 2010) method that aligns five dimensions of child development along a single measurement scale, permitting intuitive interpretation of the results. Third, we that show how our formative assessment produces actionable evidence for various stakeholders.

2. Perspectives or theoretical framework
Stakeholders at multiple levels (local, regional) are interested in measuring child development and early learning. For such child-level information to be accurate and useful, it is important that the measure provides: (1) domain-specific developmental information for each child that can be useful to teachers; including the ability to describe children’s current levels of development and to identify areas where children can benefit from additional support; (2) a quantitative score, with respect to a rigorously-defined criteria for aggregate use, that reflects a state’s progress in supporting children’s development (i.e., criterion zone boundaries [CZB]); (3) the ability to track change over time by establishing a common scale across time points; and (4) the provision of results in a reasonable time-frame to support decisions at all levels. The DRDP instrument achieves these goals using advanced psychometric methods and software for collecting and delivering data.

3. Methods and Data Sources
We analyze longitudinal data from the DRDP assessment (Fall 2015 – Spring 2017) with a multidimensional Rasch item response model that allows us estimate each child’s domain specific location at each time point. We use linking/equating techniques to track each child’s progress over time and obtain a developmental profile for each child. Additionally, we use DDA to align the dimensions. Without alignment, different dimensions cannot be directly compared. This type of direct comparison is important for communicating the results of multidimensional IRT models.

4. Results
We generated findings from a five-dimensional model consisting of 44 items. The analysis shows reasonable fit for all but four of the items. The reliability was 0.96 – 0.99 for the five dimensions. The disattenuated correlation between the dimensions was in the range of 0.91 – 0.97.

In the paper, we will briefly display and interpret technical information before shifting to a discussion of how the results provide actionable, diagnostic information. First, we will show how stakeholders use results at the individual level to (a) inform parents and teachers what the child should be working on next and (b) track student development over time. Then, we will show how aggregate results can be used to, for example, evaluate programs or show areas of need.

5. Scholarly Significance
This proposal advances an innovative and rigorous assessment system for measuring a complex (multidimensional, developmental) construct. DRDP is a theoretically grounded, authentic formative assessment and the psychometric methods (MIRM, DDA) permit analysis of development for both individuals and groups. The system permits rigorous measurement and yields actionable and diagnostic information to stakeholders at multiple levels.

Authors

• Joshua Sussman, University of California - Berkeley

• Perman Gochyyev, University of California, Berkeley