Measuring Teacher Practices at Scale: Results from the development and validation of the Teach classroom observation protocol

• In Event: Evidence-based decision making in education
  In Refereed Round-Table Session: Teachers and teacher education for equity and development

Proposal

School enrollment has increased substantially over the last 25 years in low and middle-income countries. Schooling, however, does not guarantee learning (Pritchett, 2013). A large share of children complete primary school lacking even basic reading, writing, and arithmetic skills (World Development Report, 2018) — a state of affairs UNESCO (2013) dubbed the “global learning crisis.”

The learning crisis is, at its core, a teaching crisis (Bold et al., 2017). A growing body of research indicates teaching is the most important school-based determinant of student learning (Hanushek & Rivkin, 2010; Snilstveitet et al., 2016). The difference between the impact of a weak and great teacher on student test scores has been estimated at 0.36 and 0.54 standard deviations (SDs) in Uganda (Buhl-Wiggers et al., 2017) and Pakistan (Bau & Das, 2017) respectively, which is equivalent to more than one additional year of schooling (Evans & Yuan, 2017). Moreover, evidence suggests several consecutive years of effective teaching can offset the learning shortfalls of marginalized students (Hanushek & Rivkin, 2006; Hanushek & Rivkin, 2010; Nye et al., 2004) and significantly improve students’ long-term outcomes (Chetty et al., 2014).

However, identifying effective teaching is not easy. Research shows that teacher characteristics such as formal education, years of experience (beyond the first two), IQ, and entry exam performance scores only explain a small fraction of the variation in student learning (Staiger and Rockoff, 2010; Araujo et al., 2016; Bau & Das, 2017; Cruz-Aguayo et al., 2017). Instead, recent evidence highlights the crucial role that teachers’ pedagogical skills play in explaining variation in student learning. For example, in Ecuador, Araujo and others (2016) find that, after accounting for measurement error, a one SD increase in pedagogical skills is associated with a 0.18 SD increase in learning outcomes. Similar results are found in the United States (Hamre & Pianta, 2001; Kane et al., 2011). Furthermore, Kraft and others (in press) reviewed over 40 studies on coaching programs that advance teachers’ pedagogical skills to increase student learning. They find that these interventions improve both teachers’ pedagogical skills (0.58 SD) and student learning (0.15 SD).

Although the evidence suggests teachers’ pedagogical skills are needed to tackle the learning crisis, most education systems in low and middle-income countries do not regularly monitor them, either because they don’t recognize its importance or they don’t know how. This is exacerbated by a prevalence of ineffective professional development programs, which tend to be theory-based and rarely monitor or provide actionable guidance for teachers to improve their practice (Popova et al., 2018). However, even when pedagogical skills are monitored, the instruments used to capture teacher practices fall short on several accounts, as they: i) are not evidenced-based, ii) only measure the occurrence of a given practice, rather than the quality of said practice, iii) do not meet basic reliability criteria (Ladics et al., 2018). Without a reliable instrument, even experienced education professionals struggle to distinguish between effective and ineffective teachers (Strong et al., 2011).

Given this reality, what can be done? Teach, a new open source classroom observation tool, has been developed to address these challenges. Teach has three key attributes that distinguish it from other available tools. First, Teach is the first tool to holistically measure what happens in the classroom. It does so by considering not just time spent on learning but, more importantly, the quality of pedagogical practices. This includes those practices that stimulate both children’s’ cognitive and socioemotional skills. Second, Teach has an evidence-based design and can be contextualized for use in different settings – both in terms of country-specific training materials and the ability to add new elements based on need. Third, Teach comes with a host of complementary training materials and resources that make it easier to adopt, even in low-capacity settings.

Before the tool was launched, Teach underwent a rigorous development and validation process over a two-year timeframe. A Technical Advisory Panel provided extensive feedback and inputs on the tool’s design. Teach was also piloted in over a thousand classrooms across Mozambique, Pakistan, the Philippines, and Uruguay, and has been tested with global video footage from 12 low- and middle-income countries.

This paper presents a psychometric analyses of Teach pilot data following the approach proposed by Jensen et al (2018). Results indicate Teach scores have high levels of inter-rater reliability and items provide information to differentiate among similar, but not equal, teachers' pedagogical skills. Moreover, teachers who display effective teacher practices, as measured by Teach, are associated with students that achieve higher learning outcomes.

Author

• Syeda Farwa Fatima, University of Pennsylvania