Psychometric Model for Diagnostic Classification for Multiple-Choice Option-Based Scoring: Application to a Diagnostic Classroom Assessment Instrument

• In Event: Diagnostic Assessment: Recent Advances From Psychometric Modeling to Classroom Applications

Abstract

Objectives

A new family of generalized diagnostic classification models (DCMs) for multiple-choice (MC) item assessments has been developed, denoted by GDCM-MC. It more fully captures diagnostic information by modeling incorrect options students choose. Using this model, practitioners can achieve better information regarding (1) quality of individual items (including quality of each MC option), (2) proper student diagnosis, and (3) diagnostic reliability of the instrument. In particular, the Diagnostic Geometry Assessment (DGA) has been developed to use MC items to diagnose student misconceptions. The application of the GDCM-MC model to DGA response will be discussed in this paper.

The DGA exhibits strong validity evidence for diagnosing student misconceptions using MC-option-based scoring. Thus, the DGA is perfectly aligned with the GDCM-MC model. Because the DGA was developed independently of the GDCM-MC, the application of GDCM-MC to the DGA data provides a unique opportunity to investigate the efficacy of the model in augmenting the performance of the DGA. Specifically, the objectives of the current paper are to 1) describe the application of the model to the DGA; 2) compare model-based results with classical statistical results; 3) demonstrate how model-based results provide greater detail regarding DGA performance; and 4) provide a foundation for further development of the DGA and similar assessments.

Theoretical Framework

DCMs, such as GDCM-MC, are essentially a merging of latent class models with more traditional IRT models (Rupp, Templin, & Henson, 2010). The GDCM-MC is an extension of the reparameterized unified model (Roussos, DiBello, Stout, et al., 2007).

Methods

The model’s governing equations are defined in this paper, and its parameters are described with emphasis on their interpretation in practical situations. The development of the DGA is briefly reviewed. The synergy between the model and the DGA is discussed, especially the alignment of the DGA to the theoretical assumptions of the model. Model fitting procedures are also described.

Data Sources

Forty-six teachers administered the DGA to over 1,400 students, as described in the previous presentation.

Results

Estimated item parameters, student classification statistics, and estimates of the reliability and accuracy of the classifications will be provided and compared with classical statistics. Of particular interest is the comparison of model-based student classifications to raw-score-based classifications. A detailed evaluation of the diagnostic quality of the items will be presented, along with a comparison to less detailed classical statistics.

Significance

While DCM’s have undergone extensive development over the past 20 years, the models only rarely have aligned well with the data to which they are applied. The development of the DGA and the independent development of the GDCM-MC provides a distinct opportunity in that (1) the assessment is purposely designed to be diagnostic, (2) the assessment has been empirically validated to a thorough degree (ensuring it provides accurate diagnostic classifications), and (3) the assumption of MC-option-based misconceptions is the identical assumption underlying the psychometric model. The confluence of these two areas of research has the potential to yield significant progress in classroom diagnostic assessment.

Authors

• Louis V. DiBello, University of Illinois at Chicago

• Robert Henson, University of North Carolina - Greensboro

• William F. Stout, University of Illinois

• Louis A. Roussos, Measured Progress