Measurement Invariance and Generalizability of Instructional Quality Observer Ratings Using Observation Systems in Mathematics Classroom

• In Event: AERA Roundtable Session 5
  In Roundtable Session: Classroom Observation Manuals as Measures of Teaching Quality: Possibilities and Pitfalls (Table 9)

Abstract

Objectives:
Based on a theoretical framework of instructional quality, we developed an observational instrument with which generic and subject-specific dimensions were measured in two samples of German secondary mathematics classrooms. As a step towards validating our assumptions on the generalizability of observer ratings, we combine different statistical methods to explore the psychometric properties of the observational instrument.

Theoretical framework:
Following the TIMSS video study, German educational research brought up a generic framework of instructional quality with Three Basic Dimensions (TBD, Praetorius, Klieme, Herbert, & Pinger, 2018). The three dimensions are Efficient classroom management, potential for Cognitive activation and Student support, and they are widely regarded to have a positive influence on students’ achievements in the classroom (Nilsen & Gustafsson, 2016; Scheerens, 2014). However, in a literature review, Charalambous and Praetorius (2018) gather empirical evidence that subject-specific and generic dimensions explain more variability in students’ learning outcomes than generic dimensions alone. They argue that it can be assumed that generic and subject-specific measures can provide additional value over and above each other (see also Berlin & Cohen, 2020).

Methodology:
We collected data from 76 mathematics teachers in years 7 to 10. For each teacher, we observed two lessons, which lasted 90 minutes and were spread over approximately two weeks. Extensively trained observers performed four ratings per lesson. Instructional quality was rated using the TBD observation instrument capturing three generic dimensions, as well as two subject-specific dimensions, with 26 items. They were rated on a four-step rating scale (1 = "not applicable at all" to 4 = "completely applicable"). Rater agreement was high across all items (ICC > .70). Internal consistency in terms of Cronbach’s was moderate to high for all dimensions.

Results:
For classroom management, student support and the two subject-specific dimensions, at least partial scalar invariance was found. For cognitive activation, however, there was a lack of measurement invariance already on the configurable level. As for the generalizability analysis, substantial amounts of variance could be explained by variability due to differences between teachers or classrooms. Subject-specific items showed larger amounts of lesson variability overall, whereas generic items varied more within lessons, i.e. teachers would employ more variation in classroom management and/or student support during their lessons than across different lessons.

Significance:
The results indicate that psychometric properties can differ not only between observational instruments, but sometimes even within a single instrument. In our study, we found differences in generalizability between subject-specific and generic dimensions. We argue that this finding could be relevant for the development of future observational instruments, both in research and practice. On the one hand, a lack of generalizability will have an impact on relations with other variables (e.g., student achievement). On the other hand, if decision-making in instructional practice should be based on such observations, measures need to be suitable to take such decisions.

Author

• Armin Jentsch, University of Cologne