The Diagnostic Geometry Assessment System: Results From a Randomized Controlled Trial

• In Event: Diagnostic Assessment: Recent Advances From Psychometric Modeling to Classroom Applications

Abstract

Objectives

The Diagnostic Geometry Assessment (DGA) uses cognitive research in a formative assessment system for teachers to identify and address three middle-grades geometric misconceptions. Items use a distracter-driven approach, containing the correct response, responses representative of misconception, and other incorrect responses. For each misconception, the DGA contains pre- and post-assessments, diagnostic reports, and instructional activities targeting the misconception.

This paper will present results of a randomized controlled trial to evaluate effects of the DGA on student knowledge. The paper will address the following for each misconception: How does access to DGA components affect student knowledge and misconception?

Framework

The DGA is based on theories that formative assessment is critical to instruction (Black & Wiliam, 1998; Pellegrino, Chudowsky, & Glaser, 2001). Effective formative assessments must be timely and actionable (used to change instruction to better address student needs) (Pellegrino, Chudowsky, & Glaser, 2001; Popham, 2006; USDE, 2010). The DGA uses distracter-driven assessment, which has successfully measured cognitive levels and misconceptions in science (Briggs, Alonzo, Schwab, & Wilson, 2006; Hestenes, Wells,& Swackhamer, 1992; Hunt & Minstrell, 1994; King, Gardner, Zucker, & Jorgensen, 2004; Minstrell, 1992, 2000, 2001; Minstrell & Stimpson, 1996; Sadler, 1998).

The DGA targets misconceptions that are researched in cognitive science. This paper will provide an overview of research related to the misconceptions, which are consistent with cognitive framework of misconceptions as flawed and productive thinking that should be refined and reorganized, not replaced (Smith, diSessa, & Roschelle, 1993).

Methods

Teachers were randomly assigned to varying levels of access to DGA components (diagnostic reports and instructional activities). Teachers administered pre-tests, instruction, and post-tests (assessment reports and instructional activities dependent on group assignment). ANCOVA explored the relationship between access to DGA components and post-test scores, accounting for pre-test scores. Because students were nested within teachers, a two-level hierarchical-linear-model (HLM) evaluated the relationship between teacher’s group assignment and students’ post-test scores, accounting for pre-test scores.

Data

Forty-six teachers completed the study, administering tests to over 1,400 students. The paper will provide demographics and descriptive test statistics.

Results

The paper will provide complete results of multiple ANCOVA and HLM analyses for each misconception. To summarize, ANCOVA revealed that access to any DGA component had statistically significant and practical effects on post-test correct responses for all students and post-test misconception responses for students reasoning with the misconception. HLM revealed that students reasoning with the misconception whose teachers had access to instructional activities had significantly fewer post-test misconception responses compared to students whose teachers did not have access.

Significance

This paper and the author’s colleagues’ previous work in algebra are the first explorations of distracter-driven assessment in middle-grade mathematics. From a theoretical perspective, this contributes to the growing body of research on the effectiveness of distracter-driven assessment and expands to a new subject and grade. From a practical perspective, it is imperative to provide formative assessments that are effective at improving instruction and, ultimately, student outcomes. The paper supports the use of distracter-driven assessments to individualize instruction with the goal of improving student learning.

Author

• Jessica Masters, Measured Progress