Bringing the Theory and Measurement Into Alignment: Classroom Observation Systems

• In Event: The Affordances and Limitations of Different Analytic Models for Data From Classroom Observation Systems

Abstract

Teaching is at the core of schooling and educational research should directly study it (Ball & Forzani, 2007). Classroom observation systems (Bell et al., 2018) are an important tool for measuring teaching quality (Klette & Blikstad-Balas, 2018). Most research on observation systems has focused on identifying what should be measured (e.g., Praetorius et al., 2018). There is a need to simultaneously consider how we measure teaching quality, as common statistical approaches (e.g. average scores, factor analyses) can be inconsistent with theoretical understandings of teaching quality. We highlight three common assumptions about teaching and discuss how they are violated by common measurement approaches. In violating these assumptions, observation scores may not represent the intended construct.
Assumption 1: Teaching is a goal directed activity. Teachers set and pursue specific instructional goals (Danielson, 2007), engaging in specific actions to achieve specific goals. When specific action support specific learning goals, the alignment between action and goal is a key component of teaching quality. Most measurement approaches, however, focus on average scores (e.g., Kane et al., 2012), which are a poor proxy for alignment of action and goal. In fact, many observation systems are not well structured to examine the alignment of goal and action. For example, the common approach of scoring equal-interval segments is fundamentally an approach to estimate average levels of quality.
Assumption 2: Teaching requires attending to multiple different levels. Teachers must attend to stable features of the classroom (e.g., routines, climate), features of the lesson (e.g., overall lesson goals/flow), and aspects of specific instructional interactions (e.g., responding to questions). Most measurement models, however, implicitly assumes that all measured constructs are at the same level (i.e. since the same model is used for all dimensions). For example, the Classroom Assessment Scoring System (CLASS) typically aggregates data to the classroom level as a first step in analysis, assuming that all measured constructs are stable features of the classroom (i.e. at the classroom level; Hamre et al., 2007).
Assumption 3: Teaching varies systematically based on contextual features of lessons. Teachers organize lessons within specific school, community, and curricular contexts and these contexts impact instruction. For example, the instruction just before standardized testing occurs is often unlike instruction during other periods of the year (Plank & Condliffe, 2013) while the first few days of a school year are a sensitive period for establishing routines and cultures (Bohn et al., 2004). Research on the generalizability of observation scores across the school year, however, assumes all possible observed lessons are effectively equivalent and focuses only on the number of days that must be observed to reliably generalize scores (Praetorius et al., 2014).
Many measurement practices used with observation systems are at odds with common assumptions regarding the nature of the construct that observation systems are designed to measure. Observation system developers must clearly specify the nature of teaching quality that their system captures and provide guidance for aligning measurement practices. This is vital for aligning measurement and theory, which will support deeper understandings of teaching quality.

Authors

• Mark White, University of Oslo

• Jennifer Maria Luoto, University of Oslo

• Kirsti Klette, University of Oslo