What We've Learned About Learning Progressions, Items, and Scoring Guides From Using Item Response Models

• In Event: Challenges of Assessing Learning Progressions in Science: Item Design, Validation, and Measurement

Abstract

This presentation focuses on the development and analysis of assessments for an empirically-based learning progression framework for transformations of organic carbon-containing materials at multiple scales, from atomic-molecular to global carbon cycling (Author et al., 2012). In particular we focus on accounts (explanations and predictions) of carbon-transforming processes (i.e., photosynthesis, cellular respiration, combustion, digestion, and biosynthesis).
The development of this learning progression and the associated assessments has taken place over three consecutive NSF-funded projects, and has involved multiple trials and revisions of the learning progression, the associated items, and the scoring guides and processes used for those items. We have developed several different item types, including ordered multiple choice items, sets of multiple true-false items, and short written explanation items (often, but not always, associated with one of the forced-choice item types).
This presentation will focus on the use of item response models; specifically, partial credit models (Wright & Masters, 1982) and multidimensional extensions of these models, to analyze the data from these multiple trials. Item response models allow us to evaluate the overall difficulty of items, the difficulty of transitioning from one learning progression level to another within each item, as well as the overall fit of each item and each person to the model. By using such models, we have concluded that ordered multiple choice items are not an effective item type for our particular learning progression. We have concluded that items addressing the tracing of matter, and the tracing of energy, in fact function as a single dimension. We can use graphical methods (called Wright Maps) developed for use with item response models, to show that students use accounts at similar learning progression levels across a wide variety of contexts (including plants, animals, decay, burning, and large-scale questions).
In addition, we have amassed evidence that explanation-type items are more effective than multiple true-false items, especially for identifying students at the highest level of the learning progression. Finally such models have allowed us to identify issues which make some items more or less difficult than expected, including too much or too little scaffolding, as well as items which are not actually effectively addressing the learning progression.

Authors

• Karen L. Draney, University of California - Berkeley

• Jennifer H. Doherty, University of Washington

• Charles W. Anderson, Michigan State University

• Jinho Kim, University of California - Berkeley