Value Added for Teacher Evaluation: Aspects of Interpretation and Use

• In Event: 29.046 - Administrative Sense-Making in Assessments of Teaching

Abstract

Value-added models of student achievement growth are now being widely used in schools. Researchers have examined issues associated with scoring and generalization inferences for VAM, exploring issues of bias, classification error, and other factors (see, e.g., Lockwood et al., 2007; McCaffrey et al., 2009). But an emerging perspective in validity theory argues that the chain of inferences making up the validity argument for any measure must attend to how such measures are implemented in real world settings (Kane, 2013). The present study is framed with this argument in mind, focusing on the role that school administrators play in understanding and interpreting VAM and using it in their decision-making, including teacher evaluation. Our research question is: How do school administrators make sense of VAM results and how does that influence the validity of VAMs for various purposes, particularly as an indicator of teacher quality?
We conducted an exploratory case study aimed at theory generation in a single large district in North Carolina. Our theoretical approach derives from contemporary validity theory, which seeks to trace out the full chain of inferences that make up a validity argument, and data use in complex organizations. These two lines of theory are just now coming together (see Moss, 2013 for this argument). If the validity of a measure inheres not only in the technical properties of the measure but also in how it is taken up and used “in the wild,” then validity studies must explore how conditions influence validity. Organizational theorists have long noted the critical role that sense-making plays in organizations (Weick, 1995), in how information is used to manage uncertainty and ambiguity (Cohen & March, 1986). Our initial approach then draws on ideas from this literature and its extensions to the contemporary interest in data use (eg Coburn, 2011, 2013). Principals serve as the key interpreters of such information and their practice becomes critical to an understanding of VAM and its uses.
We interviewed a dozen central office administrators and over 25 school principals, attended training sessions on VAM for administrators, and collected and analyzed state and local documents pertaining to VAM. All interviews were recorded, transcribed, and analyzed in NVIVO software. We have identified seven key components hypothesized to influence VAM interpretation and use (e.g., adequacy of technical interpretation, range of functions and uses, awareness of social effects); coded the data with the seven components, then described how principals’ beliefs about and understanding of value-added scores lead to particular decisions and actions which have real-world results for teachers, students and schools.
Preliminary data analysis reveals (1) a strong commitment to data use from district leadership; (2) a need for more training and mentoring for administrators, particularly with turnover; (3) varying perspectives about the strengths and limitations of VAM; (4) concerns about the uses of VAM and possible unintended consequences; and (4) considerable variability in principals’ technical understanding of VAM. Further, state and local policy conditions and contexts that influence interpretive processes have emerged as a key factor in our overall analysis.

Authors

• Gary Sykes, Educational Testing Service

• Laura D. Goe, ETS

• David Kirui

• Lauren Kotloff, Educational Testing Service