The Relationship Between Teaching Practices and Teacher Value-Added Measures on High- and Low-Stakes Assessments

• In Event: Investigating Policies and Practices That Promote Teacher Quality

Abstract

Objectives / Framework
Though many districts use teacher evaluation systems that include value-added measures (VAMs) and scores on classroom observations, questions remain about how the design of student assessments may impact these measures of teacher/teaching quality and the relationships between them. Just as teacher effects vary by student assessment (Corcoran et al., 2011; Papay, 2011), so might the teaching practices associated with achievement gains vary by the content and format of the test. This study focuses on three potentially “high-leverage” practices (Ball & Forzani, 2009)—modeling, strategy instruction, and orchestrating discourse—that research suggests are associated with achievement gains. It asks: to what extent do scores on teaching practices predict VAMs computed with different student assessments, and how does teachers’ enactment of these practices explain these differential relationships?

Data / Methods
This study draws on data from the Gates Foundation’s Measures of Effective Teaching (MET) project (Kane & Staiger, 2012), which includes videos of teaching linked to VAMs on high-stakes state assessments and “no stakes” supplemental assessments. My sample consists of 120 fourth grade teachers in a single district whose assessments were entirely multiple-choice and required lower levels of cognitive demand than the supplemental assessments (Polikoff & Porter, 2012).

I modified a Language Arts observation protocol (PLATO) to reliably score math instruction. Trained raters then scored 640 math and ELA lessons and took qualitative notes. Fifteen percent of lessons were double-scored (κ>.7). I used logistic regression to estimate teachers’ likelihood of being in higher value-added quintile as a function of PLATO scores. I then selected a stratified purposive sample of video cases to rewatch, transcribe, code, and analyze (Ryan & Bernard, 2000).

Findings
I find that strategy instruction and modeling are “high-leverage” practices for gains on the state math and language arts tests; teachers with higher scores are significantly more likely to be in highest value-added quintile than the lowest (p<.01). However, scores on the same practices are not predictive of gains on the more cognitively demanding supplemental assessments. Qualitative analyses demonstrate teachers taught test-taking strategies and modeled how to eliminate multiple-choice answers, using practices in ways that maximized their leverage on the high-stakes tests but that would not necessarily generalize to other outcomes.

Significance
These findings suggest that “teaching to the test” involves both content coverage and teaching practices. As states revamp their assessments to align with the Common Core State Standards (CCSS), this is an important time to consider how theories of instruction are reflected in both teaching evaluation systems and student achievement measures. These findings also raise concerns that defining “high-leverage” practices solely through their associations with VAMs may promote a system even more tightly aligned to performance on high-stakes tests. If our collective definition of effectiveness so heavily emphasizes raising achievement on certain standardized assessments, we risk losing sight of other important elements of what happens in classrooms.

Author

• Julie Jackson Cohen, University of Virginia