Using Classroom Observations to Assess the Efficacy of the Math for All Professional Development Program

• In Event: 32.066 - Presentation of a Mixed-Methods Randomized Controlled Trial Design and Lessons Learned (So Far)

Abstract

Classroom observations are a key component of the Math for All (MFA) evaluation and fit within a larger mixed methods approach. Observation data provide us with key insights into classroom instruction, the quality of teacher-student interactions, classroom climate, and allow us to directly compare the pedagogy of control and treatment teachers. The classroom observations as they are used in this study are descriptive quantitative data. Impact on classroom practice will be assessed using multiple mixed methods; teacher surveys, interviews, and work samples will help triangulate the observation data.
The MFA program focuses on helping teachers recognize the cognitive and developmental demands of a math-based task and the strengths and needs of each student (Moeller & Dubitsky, 2014). After considering different classroom observation protocols (content specific or content neutral? live or videotaped observations?) we chose the Classroom Assessment Scoring System (CLASS; Pianta, Hamre, & Mintz, 2012) because of its emphasis on the quality of teacher-student interactions and validity evidence.
The CLASS measures the quality of teacher‐student interactions within four domains: emotional support, classroom organization, instructional support, and student engagement (Pianta et al., 2012). Each of the domains is divided into dimensions of classroom quality. Observers typically watch a lesson for 15 minutes, taking notes on the specific behaviors they observe related to each of the CLASS dimensions. Scoring is completed at the dimension level using a 7-point scale, with the low range being a score of 1-2, the middle range 3-5, and the high range 6-7. The CLASS manual provides detailed information to help observers determine the specific score. The observer then watches the next 15 minutes and scores each of the dimensions again, repeating this cycle of observation and scoring until the end of the lesson. Lesson scores are created by averaging scores across all 15-minute cycles, and scores for teachers are averaged across lessons. Observations can be scored live or using video; we videotaped lessons when the teacher consented, and did live scoring in the cases where the teacher refused to be videotaped (but agreed to be observed).
One fourth grade teacher and one fifth grade teacher was randomly selected to be observed at each participating school; observations were conducted only if the teacher consented. Forty-two grade 4 and grade 5 teachers were observed in fall 2015, and 40 were observed in spring 2016; 30 teachers had full pre- and post-data. Pairs of trained coders rated the observations separately, then came together to arrive at a set of reconciled codes.
Results from the spring data collection are still being analyzed, but the preliminary results from the fall data indicate baseline equivalence between the treatment (n=19) and control (n=23) teachers (see Table 1). That is, there were no statistically significant differences between treatment and control teachers in the quality of teacher-student interactions. The final paper will include the coded data from the spring 2016 classroom observations, and present pre-post differences to assess whether the Math for All PD had a positive effect on teachers’ instruction.

Authors

• Teresa Garcia Duncan, Deacon Hill Research Associates LLC

• Rosemarie O'Conner, ICF International

• Jason Schoeneberger, ICF International