Modeling Ordered Multiple-Choice Items With the Attribute Hierarchy Method to Facilitate Learning Progression Classifications

• In Event: Diagnostic Assessment: Recent Advances From Psychometric Modeling to Classroom Applications

Abstract

Objectives

The Attribute Hierarchy Method (AHM; Gierl, Cui, & Hunka, 2008; Gierl, Leighton & Hunka, 2007) is applied to model responses to a novel item format: Ordered Multiple-Choice (OMC) (Briggs, Alonzo, Schwab, & Wilson, 2006). A distinguishing feature of OMC items is that they have response options linked to multiple levels of an underlying learning progression. Using empirical data, this paper examines the potential utility of the AHM as an approach for probabilistically classifying students along a learning progression.

Framework

Learning progressions (LPs) have been defined as “descriptions of the successively more sophisticated ways of thinking about a topic that can follow one another as children learn” (NRC, 2007, p. 219). A challenge in assessing students with respect to a hypothesized LP is designing tasks that can be used not just to distinguish correct and incorrect conceptions, but to diagnose conceptions that fall between. The OMC item format was designed to accomplish this by creating fixed response options that are mapped to different levels of a LP. An additional challenge is to find a diagnostic classification model that can flexibly accommodate several unique features of OMC items: floor and ceiling effects (when not all levels of the LP are available to be selected) and multiple response options at the same LP level.

Data

Sixteen polyotmously-scored OMC items were designed for a LP on Force and Motion (Alonzo & Steedle, 2009). These OMC items were administered to 1088 high school students at six schools in rural and suburban Iowa during the 2008-09 school year.

Methods

LP level classifications were made by implementing the AHM with an artificial neural network using the function neuralnet in R. The original Force and Motion LP was reconceptualized in terms of four attributes that were then linked to items through a reduced Q matrix and a subsequent expected response matrix. Simulated expected response patterns were used to train the neural network to make it possible to estimate attribute probabilities from the observed response matrix.

Results

Preliminary results indicate that estimates of attribute probabilities can be very sensitive to the choice of starting values for the neural network. Further, evaluating the fit of the attribute hierarchy is much more complex in the context of polytomously scored items relative to dichomtomously scored items. Finally, the best way to accommodate floor and ceiling effects is to conduct the analysis in two separate stages. Interestingly, it is not entirely clear that a probabilistic approach to classifying students in LP levels yields inferences about students that differ greatly from taking a simple deterministic approach based on observed item responses.

Significance

There have been very few examples of empirical research in which a diagnostic classification model has been applied to polytomously scored items, let alone items that have been linked to a LP. Just as importantly, the results from this study have been used to create diagnostic score reports that served as inputs for a more qualitative study (presented in this symposium) on how science teachers make sense of diagnostic assessment results.

Authors

• Derek C. Briggs, University of Colorado, Boulder

• Ruhan Circi Kizil, University of Colorado - Boulder

• Katie Larsen McClarty, Pearson

• Changjiang Wang, Pearson