Do We Measure What We Value? Estimating Value-Added for English Language Learners Across Content and Scales

• In Event: Unpacking Value-Added Measures of Teacher Performance: Implications for Disadvantaged Students and the Parents and Teachers Who Support Them

Abstract

Researchers and practitioners increasingly use value-added (VA) estimates to measure teacher effectiveness (Clotfelter, Ladd, & Vigdor, 2006). The growing usage of VA occurs in part because these measures help control for the non-random sorting of teachers and students into classrooms. Despite this growing prevalence—which includes utilization of VA to make high-stakes personnel decisions—many studies that use VA pay relatively little attention to the psychometric properties of the test used. In particular, researchers often take at face value that the scale on the test is interval in nature, an important assumption given VA, by definition, measures growth over time.

Measures like VA that rely on the test scale are said to be “scale dependent.” In this paper, we consider two aspects of scale dependence and their impact on estimates of VA. The first looks at scale as a matter of domain or dimension. For example, VA could be affected by this aspect of scale dependence if two tests of a similar construct like Algebra produced very different results. The second considers whether a given scale meets equal interval assumptions. In lay terms, a scale is equal interval if a one unit change at a given point on the scale is equivalent to a one unit change at any other point. Equal interval assumptions could impact VA if it is much more difficult for a student, say, at the low or high end of the scale to gain a unit than for a student in the middle of the distribution. To date, few if any studies examine whether these forms of scale dependency influences VA estimates.

One reason these potential issues have not been explored is practical. Oftentimes, researchers do not have multiple tests covering the same subject or different domains in that subject. We meet these practical challenges by examining various assessments for English language learners (ELLs), a subgroup facing acute educational challenges (Reardon & Galindo, 2009; Rumberger & Gandara, 2004). According to federal law, states must measure each ELL’s proficiency in both academic and conversational English, which involves tests of writing, listening, speaking, and multiple tests of reading. Specifically, these tests allow us to answer the following questions:

1. Are teacher VA estimates in reading for their ELL students dependent on the test being used?
2. Are teachers effective in one instructional area with ELLs like listening and speaking the same teachers who are effective in areas like reading and writing?
3. Could violations of equal interval assumptions be driving VA estimates, at least in part?
4.Are equal interval assumptions more or less important when a test is not vertically scaled?

In general, we find that VA is quite sensitive to the test used. There is a high attenuation-adjusted correlation between the two reading tests used, but VA using writing, listening, and speaking tests does not correlate highly with reading VA. We also find that mild changes to the test scale designed to test assumptions of an equal interval impact VA, though only slightly.

Author

• James Soland, Kingsbury Center at NWEA