A Diagnostic Assessment of Teachers’ Understandings of Rational Number

• In Event: Harnessing Psychometric Models to Develop Next-Generation, Research-Based Assessments of Rational Number Knowledge

Abstract

Objectives
This paper presents analysis of a national dataset collected by the Diagnosing Teachers’ Multiplicative Reasoning (DTMR) project. We created an assessment to diagnose middle grades teachers’ facility with several components of reasoning with rational number, including attending to referent units, iterating, partitioning, and forming multiplicative comparisons. These components are termed attributes. We will use diagnostic classification models (DCMs; Rupp, Templin, & Henson) to analyze teachers’ responses to the assessment and evaluate (a) how the attributes relate to each other, (b) how well items on the assessment measure the attributes, and (c) with which attributes teachers are or are not facile. More generally, this study illustrates the types of information DCMs can provide about examinees’ mathematical understandings, introducing this newer psychometric model to mathematics educators that may have interest in developing a DCM-based assessment in their own research areas.

Perspectives
DCMs can provide reliable, multidimensional feedback to students and teachers from an assessment (Templin & Bradshaw, in press). Such information mirrors the multifaceted nature of core content objectives that comprise state and national mathematics standards. Thus, DCMs offer feedback that is more aligned to educational objectives than IRT models that provide unidimensional feedback with respect to an overall ability. Because DCMs focus on measuring multiple attributes instead of an overall ability, they help identify an examinee’s strengths and weaknesses. This information can be used in turn to tailor instruction for students or professional development for teachers.

Methods and Data Sources
First, we have conducted a simulation study to evaluate the expected accuracy of model estimation under the DTMR assessment’s conditions (i.e., number of items, which attributes were measured by each item, sample size).

Second, we will analyze the DTMR data using the log-linear cognitive diagnosis model (LCDM; Henson, Templin, & Willse, 2009). The data consists of a national sample of in-service middle grades teachers’ scored responses to the 22-item DTMR assessment. The assessment has 19 multiple choice items and three open-ended items.

Results
Results will include findings from the simulation study and the empirical data analysis. A main result will be reports of teachers’ facility with the attributes at the individual and aggregate levels.

Scientific/Scholarly Significance
Previous research has retrofitted DCMs to assessment data with disappointing results. This paper presents one of the first DCM analyses of data from an assessment that was designed within the DCM framework from the onset. Although educational assessment practices demand diagnostic assessments aligned with standards, assessments are commonly not made to report multidimensional scores. This study provides an example of how this can be done in one critical content area.

References
Henson, R., Templin, J., & Willse, J. (2009). Defining a family of cognitive diagnosis models using log linear models with latent variables. Psychometrika, 74, 191-210.
Rupp, A. A., Templin, J., & Henson, R. (2010). Diagnostic measurement: Theory, methods, and applications. New York: Guilford.
Templin, J., & Bradshaw, L. (in press). The comparative reliability of diagnostic model examinee estimates. Journal of Classification.

Authors

• Laine Bradshaw, University of Georgia - Athens

• Jonathan Templin, University of Georgia

• Andrew G. Izsak, University of Georgia