Search
Program Calendar
Browse By Day
Browse By Time
Browse By Person
Browse By Room
Browse By Unit
Browse By Session Type
Search Tips
Annual Meeting Housing and Travel
Personal Schedule
Sign In
X (Twitter)
Theoretical Framework and Objectives
All measurement instruments contain a theory of teaching quality (Gitomer & Bell, 2013). When instruments are used in a study, this theory undergirds everything from the constructs measured to the analysis and reporting (Moss & Haertel, 2016). This paper provides an overview of the measurement approach for two of the TALIS Video study’s primary measures of teaching: protocols for classroom observations and classroom artefacts.
As previous papers in this session suggest, the TALIS Video study is guided by the conceptual frameworks of PISA (OECD, 2016), TALIS (OECD, 2013) and the participating eight countries/jurisdictions. These frameworks and an extensive review of the observation and artefact research define the constructs that were selected for measurement in the observation and artefact instruments. To operationalize these constructs in scalable instruments we used evidence-centered design principles (Mislevy, Almond, & Lukas, 2003) to guide our development efforts. We also considered the increasingly sophisticated literature on various measurement issues implicit in the measurement of teaching, such as video or live scoring (Casabianca et al., 2013) and how to minimize rater error (Cash, Hamre, Pianta, & Myers, 2012).
Method and Data Sources
An extensive review of the global literature on observation and artefact instruments revealed more than 1,300 non-duplicate references that were culled down to 107 references. Studies and instruments were reviewed to 1) identify constructs that were related to student outcomes and 2) review how researchers had partitioned teaching and operationalized it in their respective instruments. Concurrently, we identified the claims we hoped to make from the observation and artefact scores and specified types of evidence we would need to support those claims. We then drafted codes and tested them against videos and artefacts the collaborating countries provided. Over almost 18 months, there were four rounds of revising and testing for both sets of codes, with each round including samples of teaching from the countries and feedback from country representatives.
Results and Significance
The observation codes are comprised of two sets of codes – higher and lower inference codes – that measure six domains of teaching quality: classroom management, social-emotional support, quality of subject matter, discourse, student cognitive engagement, and assessment and responses to student thinking. The lower inference codes focus on distinct, recognizable behaviors that can often be fully observed in short periods of time (e.g., time on task, experiential connections, presence of technology), the higher inference codes focus on dimensions of teaching for which many behaviors might support a score and require more time (teacher feedback, positive climate, nature of discourse). The artefact codes focus on the same domains (dropping classroom management and social-emotional support) and include codes such as accuracy of materials, questioning, and addressing diverse student needs. All codes are applied to sample lessons from the quadratic equations unit – two lessons (observations) or four lessons (artifacts).
These codes will produce scores that describe teaching and will be analyzed in conjunction with teacher beliefs, student outcomes and contextual variables. Code quality and implementation across countries is therefore critical to the study’s success.
Courtney A. Bell, ETS
Brian Stecher, The RAND Corporation
Jonathan Schweig, The RAND Corporation
Margaret Witherspoon, Educational Testing Service
Yi Qi, Educational Testing Service
Mariana Barragan, University of California - Los Angeles