Different Frameworks, Same Goal? A Systematic Comparison of Two Observation Manuals to Measure Teaching Quality

• In Event: Breaking Out of Silos: Using Classroom Videos for Cross-Disciplinary and Cross-Methodological Examinations of Teaching

Abstract

Objective and Perspectives
Providing high-quality teaching to all students is a key to justice in education (Shields & Mohan, 2008). One way to enable teachers to provide high-quality teaching is to offer feedback on teaching quality, for example through classroom observations using observation manuals. However, different observation manuals hold to different theoretical traditions and conceptualizations of teaching quality, which can potentially hinder accumulation of knowledge and evidence about teaching quality across contexts. As such, it is necessary to understand how different manuals conceptualize and understand teaching quality (Praetorius & Charalambous, 2018). One way to address this issue is to systematically compare the results of different observation manuals applied to the same classroom teaching episodes.

Methods
In the present investigation, we applied two observation manuals to the same set of classroom video data: an observation manual based on the framework of the three basic dimensions of teaching quality (TBD; Fauth et al., 2022) and the Protocol for Language Arts Teaching Observation (PLATO; Grossman, 2015). We aim to identify overlaps and differences between the manuals’ conceptualizations and assessment of teaching quality.
The TBD framework conceptualizes teaching quality using three overarching and generic basic dimensions: Cognitive activation, Student support, and Classroom management (Praetorius et al., 2018). The TBD manual comprises 11 teaching quality aspects assigned to these three basic dimensions. Meanwhile, the PLATO manual defines four overarching dimensions of teaching quality: Instructional Scaffolding, Disciplinary Demand, Representation and use of Content, and Classroom Environment. Comprised in these four dimensions are 12 teaching quality aspects. While, originally intended as a subject-specific manual for language arts, PLATO has been utilized as a generic observation manual, applied for example in mathematics classrooms (Cohen, 2018).

Data
To conduct the comparison, certified raters for each manual, scored one double-lesson in mathematics and two single lessons in language arts, both from Norwegian lower secondary classrooms. All comprised teaching quality aspects are scored on a four-point scale in both observation manuals. However, as the TBD manual provides one score for the whole lessons, while the PLATO manual is scored in 15-minute segments, both frameworks were utilized to provide a score for the whole lesson and for 15-minute segments to enable comparisons.

Results
The findings show that despite the different conceptualizations of teaching quality, both manuals scored teaching quality similarly at an overarching level over the course of the lessons, suggesting that both manuals have similar overarching views of teaching quality. Moreover, the raters used similar events as evidence for their teaching quality scores.

Significance
Even though this study is based on a small number of classroom videos, it might serve as a starting point to compare observation manuals more systematically in the future. These results can benefit researchers and practitioners in choosing an appropriate manual for their context. Furthermore, this can improve our understanding of the possibilities and limitations of different theoretical frameworks and manuals used for classroom observations in order to provide teachers with the best feedback possible for high-quality teaching.

Authors

• Tosca Daltoè, University of Tübingen

• Alexander Viktor Selling, University of Oslo