Using Lesson and Teacher-Level Profiles to Analyze Instructional Quality

• In Event: The Affordances and Limitations of Different Analytic Models for Data From Classroom Observation Systems

Abstract

Objective: Recent work theorizes that positive and warm classroom environments may be necessary to help students meet the ambitious goals outlined in newer mathematics standards. To examine whether there was empirical evidence supporting this claim, our study explored two research questions: 1) What are the characteristics of classroom learning environments during lessons where raters do and do not observe College and Career Ready (CCR)-aligned mathematical engagement?; 2) In what ways do the classroom learning environments and students’ engagement with CCR-aligned mathematics content vary within and across teachers?
Framework: Recent research suggests that empirical profiles of instruction can help explore variability in teaching both within a teacher’s lessons, as well as between different teachers (Halpin & Kieffer, 2015; Keller et al., 2018). To that end, we used latent profile analysis to uncover groups of similar lessons, or lesson profiles. These profiles provided a way to explore characteristics of classroom learning environments in lessons where raters do and do not observe CCR-aligned mathematical engagement (Magidson & Vermunt, 2004). This analytic technique has the potential to surface the co-occurrence of particular lesson characteristics and determine whether there are patterns of co-occurrence pervasive enough that they can be used to sort lessons into different groups (Keller et al., 2018).
Data and Methods: We scored 422 upper elementary mathematics lessons from 49 teachers with (1) measures of the classroom learning environment from the Classroom Assessment Scoring System (CLASS; Pianta & Hamre, 2009), and (2) teachers’ and students’ engagement with college and career ready (CCR) mathematics content with a new tool called the Instructional Practice Research Tool - Mathematics (IPRT-M). Multilevel modeling techniques allowed us to also explore patterns of instruction at both the lesson and teacher level (Mäkikangas et al., 2018).
Findings: Four lesson profiles nested in three teacher groups emerged from the data (see Figure 1). Raters only observed consistent opportunities for CCR-aligned mathematical engagement in environments that were safe and productive, as well as positive, supportive, and engaging. The three distinct teacher groups suggest that students in different classrooms had divergent access to mathematically engaging learning environments. For 55% of these teachers (the Inconsistent and Orderly groups), observers never or only rarely saw evidence of high levels of Emotional Support or Student Engagement. In only two of the 125 lessons taught by teachers in the Inconsistent and Orderly groups did raters observe consistent CCR-aligned engagement.
Significance: We could not have uncovered either these instructional or teacher profiles without the use of multilevel latent profile analysis. At the lesson level, only a lesson-centered approach would have surfaced the important distinctions between the Orderly and Inconsistent learning environments. Methods such as multilevel latent profile analysis that identify common instructional patterns in lessons and their occurrence within and across teachers may prove especially beneficial to policymakers and district personnel seeking to better understand a particular instructional landscape or to move beyond a one-size-fits-all model of teacher support.

Authors

• Rebekah Berlin, Deans for Impact

• Julie Jackson Cohen, University of Virginia