Teachers’ Interpretations of Score Reports Based Upon Ordered Multiple-Choice Items Linked to a Learning Progression

• In Event: Diagnostic Assessment: Recent Advances From Psychometric Modeling to Classroom Applications

Abstract

Objectives

Research on teachers’ data-driven decision making has focused on interpretations of summative test scores (Swan & Mazur, 2011), rather than on results from classroom-level diagnostic assessments. In addition, new frameworks and strategies for assessment produce data that may be unfamiliar to teachers. This paper explores teachers’ interactions with diagnostic score reports produced using a novel form of multiple-choice item, linked to levels of a learning progression (LP) through application of the Attribute Hierarchy Method (AHM; Gierl, Leighton & Hunka, 2007).

Theoretical Framework

LPs—“descriptions of the successively more sophisticated ways of thinking about a topic that can follow one another as children learn” (National Research Council [NRC], 2007, p. 219)—hold promise for influencing a wide range of educational policies and practices. LPs, such as the Force and Motion LP (Alonzo & Steedle, 2009) used in this study, can serve as a cognitive model for assessment development (NRC, 2001).

Ordered multiple-choice (OMC; Briggs, Alonzo, Schwab, & Wilson, 2006) items are an efficient means of assessing student understanding relative to a LP. However, little is known about how teachers interpret scores based upon these items, particularly when scores are derived using novel psychometric techniques such as the AHM (see previous presentation).

Data Sources

Seven high school physics teachers participated in the study. Each was recommended because of expertise in formative assessment.

Methods

Three video-recorded interviews were conducted with each teacher. The first interview focused on the teachers’ current formative assessment practices. The second and third interviews were “think alouds” (Ericsson & Simon, 1993) as teachers interacted with a set of paper-and-pencil score mock score reports and an interactive computer-based score report, respectively. The latter was developed by applying the AHM to real student data (see previous presentation).

Using grounded theory coding (Charmaz, 2000), our codes captured a) teachers’ current formative assessment practices in the first interview and b) teachers’ interactions with the score report(s) in the second and third interviews.

Results

Specific points of confusion or misinterpretation indicate where teachers struggled with LP based score reports. For example, some teachers viewed probabilities of level mastery as a percentage of correct items at a given level. These teachers struggled to interpret the probabilities, since all levels were not necessarily represented in each OMC item. In addition, we found congruency between teachers’ existing formative assessment practices and their use of the LP based score reports. For example, teachers who viewed existing data item-by-item found information about individual items to be meaningful, seeing less value in LP diagnoses.

Significance

As psychometric and assessment techniques become more sophisticated, it is crucial to consider how the results will be used in real classrooms in order to inform the design of professional development opportunities and data displays. While not surprising, given research on educational reforms (e.g., Fullan, 2000), our results points to the importance of supporting teachers (through professional development and/or different data displays) to think differently about new forms of data; otherwise, diagnostic information may be misinterpreted.

Authors

• Alicia C. Alonzo, Michigan State University

• Xeng Elizabeth de los Santos, California State University - Long Beach

• Jennifer L. Kobrin, Pearson