Diagnostic Measurement Models for Item Response Dependencies Caused by Misconception Effects

• In Event: Diagnostic Assessment: Recent Advances From Psychometric Modeling to Classroom Applications

Abstract

Objectives

This paper presents a psychometric framework aligned with learning theories that drive the development of concept inventories (Hestenes, Wells, & Swackhamer, 1992): namely, theories that misconceptions exist and systematically influence item responses. The framework leverages a class of multidimensional psychometric models, diagnostic classification models (DCMs) (Rupp, Templin & Henson, 2010), to capitalize on information in students’ incorrect answers to provide classification-based, reliable diagnoses of latent misconceptions. This study will introduce specific model parameterizations and illustrate types of feedback that different parameterizations can provide to students.

Perspectives

Concept inventories, or distracter-driven assessments (Sadler, 1998), are multiple choice tests designed to identify misconceptions that students’ possess. On these tests, the incorrect options (i.e., the distracters) are common erroneous answers reflective of known student misconceptions. Thus, which incorrect option(s) a student selects provides evidence for which misconception(s) the student possesses. Common methods of scoring for concept inventories—total sum scores for misconceptions—may introduce measurement error by not allowing item options to contribute differentially to the misconception measure and by interjecting an unquantifiable source of error: the cut-off point to determine the score above which indicates the diagnosis that a student possesses a misconception. To address these limitations, the methodology developed in this proposal uses a latent variable model framework where misconceptions are estimated as Bernoulli-distributed, random effects in the item response.

Methods

Methods will extend the work of Bradshaw and Templin (2013), which developed the Scaling Individuals and Classifying Misconceptions (SICM) model. Variants of the SICM model will be developed that estimate a latent ability with several discrete distributions, instead of a continuous distribution as in Item Response Theory.

Data Sources

A simulation study will be conducted to evaluate the accuracy of model estimation under various testing conditions (i.e., number of items, number of options measuring misconceptions, sample size). Then, Force Concepts Inventory (FCI; Hestenes et al., 1992) data from 10,039 high school students will be analyzed.

Results

Results will include findings from the simulation study and the empirical data analyses. A main result will be illustrating reports of students’ misconceptions at the individual and aggregate levels and comparing the various model parameterizations.

Scientific/Scholarly Significance

This study provides one framework of psychometric methodology that is useful for providing validity evidence for inferences about misconceptions based on concept inventory results. This type of evidence is needed to ensure appropriate use of the test results for informing teaching and learning.

Author

• Laine Bradshaw, University of Georgia - Athens