Understanding (dis)agreement in student ratings of teaching and the quality of the learning environment

• In Event: Student feedback on teaching in schools

Abstract

Classroom climate plays a critical role shaping school experiences (Cohen et al, 2009). Positive classroom climates promote student achievement (Kane & Cantrell, 2010), motivation (Tuan et al., 2005) and self-efficacy (Fast et al., 2010). Student surveys are increasingly used to monitor and improve classroom climate. These surveys have strong face validity because students are uniquely qualified reporters, having extensive knowledge of their teachers and classroom interactions (Ferguson, 2012; Downer et al, 2014).
Climate measures are created by aggregating student responses to create classroom ratings. Student responses are typically aggregated regardless of how much ratings vary within classrooms (Chan, 1998). This presents an issue in that classrooms with different response patterns can yield the same aggregate rating. However, within-classroom variation in student responses could reflect differences in student experiences. Variation may reflect teacher differential expectations for subgroups of students, or efforts to accommodate diverse student needs (Freeman & Crawford, 2008). Lastly, climate perceptions could vary based on student characteristics (Garza, 2009).
This study addresses the following questions:1) To what extent does instructional format and the nature of student-teacher interactions vary across classrooms? 2) In what ways are classrooms with low student consensus different than classrooms with high student consensus?
From 2009—2012, the MET study collected teaching and learning data from over 3,000 K-12 teachers across the U.S. We used a subset of 13,945 students in 554 math and science classrooms. To assess classroom climate, the Tripod (Ferguson, 2010) was used, which assesses seven climate dimensions—Caring, Captivating, Conferring, Clarifying, Challenging, Controlling, and Consolidating. Items use 5-point scales (1 = totally untrue and 5 = totally true). Value-added models on student achievement measures were used to obtain estimates of teacher effectiveness (Kane et al. 2013). We measured item within-classroom variability using r_wg, which indexes shared climate perceptions within organizations (James et al, 1984). Moreover, explored the relationships among within-classroom consensus, student characteristics, and observation-based measures of instructional practice using hierarchical linear models.
The results indicate that classrooms show low levels of consensus for many items, indicating that students in the same classroom do not agree in their climate perceptions. We also find that climate perceptions are associated with student characteristics. Female and low-achieving students give systematically different responses than their male and high achieving classmates, respectively. Moreover, classrooms with lower levels of consensus tend to have higher observation ratings of instruction. This may reflect deliberate differentiation by teachers to meet individual student needs (Gregory & Chapman, 2012). It may also be the case that students internalize the same classroom stimuli in different ways (Seidel, 2006).
This study investigated the role that within-classroom consensus may play in using student surveys to understand classroom climate. The results have implications for how student surveys are used in formative and summative evaluation. The recognition that consensus may be substantively interpretable can inform our thinking about the inferences that may be appropriately derived from student survey data, and how best to use that information to provide feedback to teachers.

Authors

• Jonathan Schweig, The RAND Corporation

• Jose Felipe Martinez, University of California - Los Angeles