Instructional Quality Traveling Across Contexts: Possible Biases in Conceptualizing Instructional Quality in Observation Systems

• In Event: AERA Roundtable Session 5
  In Roundtable Session: Classroom Observation Manuals as Measures of Teaching Quality: Possibilities and Pitfalls

Abstract

Objective:
Observation systems is today popular in cross-national classroom video studies (Praetorius et al., 2019). However, conceptualizations of instructional quality embedded in observation systems often derive from American and central European contexts (Praetorius & Charalambous, 2018), while we know little about conceptual and methodological biases when applied across different contexts. This study discusses possible biases in a standardized system (the Protocol for Language Arts Teaching Observations [PLATO] (Grossman, 2015) of instructional quality when applied across a sample of Nordic mathematics and LA classrooms. I especially identify three aspects of possible biases (related to the grain size of conceptualization, operationalization, and sequencing of lessons) and how these may influence ratings of instructional quality when using PLATO in a Nordic context.

Framework:
To discuss possible biases, I differentiate among construct, method, and item bias based on a framework for cross-cultural studies (van de Vijver & Tanzer, 2004). I discuss how such possible biases relate to context-sensitivity—the extent that the operationalization of instructional features are broad enough to capture somewhat different instructional practices of the same construct (Praetorius et al., 2019), with empirical examples of scoring using the PLATO manual.

Data and Methods:
Drawing on PLATO’s 12 elements of instructional quality (grouped in four different domains), the three elements Intellectual challenge, Uptake of student responses, and Feedback in mathematics lessons in Norway and Finland (N = 47) were purposefully chosen to illustrate particular issues in coding. To analyze possible biases in scoring these elements, I studied the rubrics, the scoring, the transcripts, and scoring justifications along with videos. This was an iterative process and did not result in any statistical reports on potential biases, yet enabled a theoretical and methodological discussion of possible biases in observation systems when applied in different contexts.

Results:
PLATO seem to favor some activity formats above others, and especially how individual seatwork is defined and categorized had implications for measuring instructional quality with all three PLATO elements. The way Intellectual challenge is conceptualized does not acknowledge individual seatwork nor differentiated instruction, common for mathematics instruction in Nordic contexts, which resulted in challenges in scoring. Similarly, Uptake of student does not include individual seatwork situations in which teachers use uptake in discussing content with individual students. Finally, lessons structured as individual seatwork tend to have Feedback instances spread out instead of clustered within segments. I discuss implications for scoring and whether these issue may introduce ‘bias’ in measuring instructional quality (van de Vijver & Tanzer, 2004).

Significance:
The ways teachers organize lessons might cause issues for any analysis of classroom instruction. I argue that awareness and transparency of possible biases is especially necessary for valid interpretations of ‘instructional quality’, and information of biases might improve observation systems so they facilitate comparable data and knowledge aggregation across contexts. Additionally, problematizing the use of observation systems in different curricular and educational contexts might help to encourage a more nuanced perspective of instructional quality that is not solely based on research from a specific scholarly tradition.

Author

• Jennifer Maria Luoto, University of Oslo