Evaluating Teachers in the "Grand Experiment": How Organizational Context Shapes Policy Responses in New Orleans

• In Event: Exploring the Implementation and Effects of Various Levers in Multiple-Measure Teacher Evaluation

Abstract

Multiple-measure teacher evaluation systems have rapidly gained popularity in states and districts nationally. While many studies focus on the reliability and validity of evaluation measures (e.g., Hill, et al., 2012; Chetty, et al., 2014; Ho & Kane, 2013), there is a dearth of research on the implementation of such reforms, and more specifically the ways in which organizational context shapes implementation. New Orleans provides a strategic case to examine how a state teacher evaluation policy (called Compass) is enacted in a highly decentralized local setting with wide variation in organizational context. Utilizing a multiple case-study approach, we collected documents and conducted interviews with system leaders (n=3), school administrators (n=17), and teachers (n=35) in eight case study schools and their associated governing organizations (i.e., districts, charter management organizations). Cases were selected to reflect diversity in school governance (traditional, independent charter, networked charter) and level (elementary, middle, high). To frame our analysis, we adapt Jennings’ (2012) typology of five features of accountability systems, examining how these classifications relate to school organizational characteristics and ultimately influence the implementation of evaluation policy.

First, we ask, how and to what extent does the implementation of state-driven evaluation policy vary across school settings? Preliminary analyses suggest a range of responses to the Compass policy, from productive to compliant to distortive. Among the cases, one outlier school demonstrated a productive responses to the policy (e.g., practices that promote reflection and teacher learning), one a hybrid compliant/productive response, three a mix of all three types of response with a focus on compliance, and three others only compliance and distortive responses (e.g., using strategic behaviors like cancelling observations that were not going well and removing students having “bad days” from the classroom during observations).

Next, we ask, what organizational factors contribute to the variation in implementation across school settings? Preliminary findings from our cross-case analysis indicate that leadership, capacity, and culture appeared to mediate implementation. That is, transparent, shared leadership facilitated more productive forms of implementation, while directive, top-down leadership led to more compliant responses and occasionally to distortive responses. We also observed that some leaders acted as buffers to the policy, by obtaining state policy waivers or tailoring observation protocols, while other principals acted as enforcers of standardized compliance. Further, high individual and organizational capacity, including additional staff to support the process, past experience implementing similar policies, and system-level guidance (from district and/or charter network) facilitated productive implementation. Interestingly, autonomy did not appear to make a difference to the kinds of policy responses observed in our case schools. Finally, professional culture focused on reflection and learning was crucial to productive responses to the policy.

This paper offers a substantive contribution to the present discourse on teacher evaluation policy. First, we offer a useful framework for classifying and understanding evaluation systems and implementation responses. Second, we draw upon New Orleans’ unique experiment in school choice to identify organizational factors that shape responses to policy. In conclusion we offer recommendations for policy, practice and future research.

Authors

• Julie A. Marsh, University of Southern California

• Katharine Omenn Strunk, University of Southern California

• Susan C. Bush-Mecenas, University of Southern California

• Jane Arnold Lincove, Tulane University

• Alice Huguet, Northwestern University