Lessons Learned About Videotaping and Coding Classroom Observations

• In Event: Math for All: Lessons Learned From Piloting a Randomized Controlled Trial in a Large Urban District

Abstract

Classroom observations are a key component of the MFA study. Observation data provide us with key insights into classroom instruction, the quality of teacher-student interactions, classroom climate, and allow us to directly compare the pedagogy of control and treatment teachers. The benefits of a pilot have been tremendous, and we believe we have many key lessons learned to share with fellow researchers.
Lessons learned from the pilot
• Selecting an observation protocol. We had originally proposed using the Mathematical Quality of Instruction (MQI) protocol. The MQI provides separate scores for various elements of effective mathematics teaching in three different areas: teacher-content relationship; teacher-student relationship; and student-content relationship. The MQI scoring protocol was designed for assessing videotaped mathematics lessons. Each lesson is divided into five- to seven-and-a-half-minute segments for scoring. Raters assign each segment a score for each of the MQI elements, and also assign an overall score to the whole lesson. Each lesson is scored by two raters working independently, and scores are averaged across lessons to derive a teacher score (MET Project, 2010).
The MQI seemed a perfect fit because of its focus on content. But the MFA intervention strongly emphasizes differentiated, individualized instruction, and is less about math content than it is about enhancing teachers’ awareness of and ability to tailor instruction to a student’s needs. Accordingly, we looked into using the CLASS (Pianta, Hamre, & Mintz, 2012) as our observation protocol. The CLASS measures the quality of teacher‐student interactions within four domains: emotional support, classroom organization, instructional support, and student engagement. Each of the domains is divided into dimensions of classroom quality. Observers rate classrooms based on the twelve dimensions, scoring each dimension on a 7‐point scale.
The MQI versus CLASS decision was not one that could be made solely on the basis of content versus non-content focus; we also had to consider mode, inter-rater reliability, and project budget.
• Videotaping versus live coding. Videotaping gives us the ability to re-watch, but live coding allows the coder to directly experience what happens during the math lesson. But videotaping makes teachers self-conscious and some are fearful that the video might be used for teacher evaluations. Videotaping also requires equipment and technological savvy on the part of the classroom observer, to ensure quality of video and audio.
• Achieving acceptable inter-rater reliability. The MQI offers an online training that takes upwards of 16 hours, at the end of which one takes a certification test. Math content knowledge is assessed prior to the online training. The CLASS requires coders to take a two-day, in-person training, after which an online certification test is given. We found that the MQI was very challenging, even for PhDs: a math education background really is a pre-requisite.
• Making hard decisions: Optimizing study rigor within a limited budget. Ultimately, we chose to do live coding with the CLASS, and to videotape 40% of the participating teachers so that we can also code the videos with the MQI, to be able to directly compare the two instruments’ results.

Authors

• Teresa Garcia Duncan, ICF International

• Rosemarie O'Conner, ICF International