Design and Implementation of a Practical Measure Focused on the Quality of Discussion in Mathematics Classrooms

• In Event: Hilton Roundtable Session 13
  In Roundtable Session: Practical Measures in Service of Instructional Improvement: Attending to Issues of Rigor

Abstract

This paper draws from a project that is developing a system of practical measures, routines, and representations to be used in five research-practice partnerships, all of which are focused on improving the quality of instruction in middle-grades mathematics classrooms. The presentation will focus on how we have attended to issues of rigor in the design and use of a practical measure specific to the quality of whole-class discussions. A substantial body of research in mathematics education has shown that whole class discussions are important for supporting students to both develop conceptual, and enduring, understandings of key mathematical ideas and for developing key disciplinary practices (e.g., Stein, Engle, Smith, & Hughes, 2008); however most whole-class discussions fall short of reaching this outcome (e.g., Boston & Wilhelm, 2015).

Given Carnegie’s finding that three-minute student surveys can be generative (Yeager et al., 2013), we created a short student survey of the quality of whole class discussion. We attended to rigor in a number of ways in the design of the measure. First, the focus of the items were grounded in a robust, existing research base. Second, we attended to face validity. For example, we anticipated that using students as informants would have face validity with teachers. And, we generated student survey items in partnership with users of the measures (e.g., district mathematics leaders, mathematics coaches). Third, we engaged in three cycles of design and analysis to test and refine the measure of whole class discussion. Each cycle entailed researchers and district math leaders observing lessons in classrooms in which the quality of whole class discussions varied. We did this to ensure that our observations matched with students’ survey responses. In addition, after each observation, we conducted audio-recorded cognitive interviews (Desimone & LeFloch, 2004) on the current survey items with approximately five students selected to represent a range in participation. Qualitative analyses of the cognitive interviews supported us to clarify both how students interpreted items and why they selected particular response options, with particular attention to items identified as problematic. In doing so, we attended to whether the items enable us to detect differences in the varying quality in instruction across classrooms.

This fall, we will gather evidence of reliabilty through repeated adminstration of the survey to the same students, complemented by outsider observations of the quality of whole-class discussions. Surveying the same students over time provides information about the consistency of student responses over time when there are no changes in instruction, and when there are changes to instruction. In addition, repeated administrations of the survey coupled with classroom observations will be used to determine the the ability of the practical measure to identify change when change occurred, and to attribute change in the practical measure to change in a particular feature of instruction that the teacher was working on.

Authors

• Marsha M. Ing, University of California - Riverside

• Kara J. Jackson, University of Washington - Seattle

• Paul A. Cobb, Vanderbilt University

• Erin Craig Henrick, Vanderbilt University

• Nicholas M. Kochmanski, Vanderbilt University - Peabody College

• Hannah Nieman, University of Washington

• Thomas M. Smith, University of California - Riverside

• June Ahn, New York University

• Anna-Lena Stift, University of California - Riverside

• Daniela Kruel DiGiacomo, University of California - Riverside

• Maria Hays, University of Washington - Seattle