Affordances and Constraints of Assessment Modes in Measuring Teaching: An Argument-Based Cross-Cultural Approach

• In Event: Measuring Teaching Practices and Teaching Quality: Conceptualizations and Findings From an International Study

Abstract

Objectives: Researchers frequently must decide how best to measure teaching quality. This paper takes a validity argument approach to understanding the affordances and constraints of four commonly used modes to measure teaching quality: teacher questionnaires, student questionnaires, teaching artifacts, and classroom observation systems. We articulate criteria against which the quality of each mode’s validity evidence can be evaluated. We apply these validity criteria to data from the TALIS Video study’s eight countries, treating each country as a repeated measurement case. The analysis illustrates the tradeoffs researchers face when selecting a measurement mode for assessing teaching quality in research contexts.
Theoretical Framework: Foundational insights about teaching and learning rest, in part, on the validity of the instruments used to measure teaching quality. Recent studies of observation systems (Campbell & Ronfeldt, 2018), teacher and student questionnaires (Fauth et al., 2017; Schenke, et al., 2018), and classroom artifacts (Joyce et al., 2018) provide insights into the validity of these instruments. Single studies frequently compare two measures, e.g., observations and student questionnaires. This allows for important substantive comparisons and parsimonious data collection. Given expenses and burden, it is uncommon to consider more than two modes of measuring teaching quality. Thus mode insights have developed across studies with different study designs and samples, making mode comparisons more complex. The TALIS Video study provides the rare opportunity to understand the relationships among four common measurement modes within a single country and across countries.
Methods and Data Sources: Drawing on a validity argument approach, we specify the inferences most critical to research purposes and then evaluate each with empirical analyses. The inferences are: the measure 1) can be implemented by country teams, 2) captures variation in the population, 3) is sufficiently reliable, 4) has approprite concurrent and predictive validity. Data come from the four aforementioned modes in eight country/economies. Validity evidence includes rater certification rates, rater agreement metrics (exact and adjacent agreement and kappas), descriptive statistics, and correlations within and across modes. The predictive relationships between each measurement mode’s instrument(s) and student outcomes (mathematical achievement, efficacy, and interest) are also modeled with cluster adjusted regressions.
Main Results: All four modes were able to be implemented in every country, although notable differences occurred across countries in teachers’ and students’ responses to specific questionnaire scales. Each mode had generally acceptable levels of variation. However, some scales/codes had extremely low levels of variation. Artifact and observation agreement rates were similar to levels achieved in national studies (60-99%). Correlations across modes within a country varied from -0.25 to +0.56. There was stronger predictive evidence for interest and efficacy outcomes compared to achievement. Artifacts had almost no predictive validity for achievement.
Significance of the Work: Participant burden, cost and efficiency concerns require researchers to carefully select measurement modes that can assess teaching quality with high levels of validity. This paper provides a rare and unprecedented validity analysis of four common measurement modes suggesting that the “best” mode varies by country, domain, and outcome of interest.

Authors

• Courtney A. Bell, University of Wisconsin/ WCER

• Katherine Furgol Castellano, Educational Testing Service

• Eckhard J. Klieme, DIPF|Leibniz Institute for Research and Information in Education

• Brian Stecher, The RAND Corporation

• Jonathan Schweig, The RAND Corporation