Implementing Teacher Effectiveness Measures Under Loosely Defined State Mandates: Practitioner Considerations at the LEA (Local Education Agency) Level

• In Event: Value-Added in Arizona: Specifics, Statistics, and Impact

Abstract

Purpose

This study investigates analytic methods that Local Education Agencies (LEAs) might utilize to construct measures of instructional effectiveness under loosely defined state mandated teacher evaluation requirements (Arizona State Board of Education, 2011). The focus is not only on evaluating the technical aspects of value-added measures (McCaffrey, Lockwood, Koretz, & Hamilton, 2003) but also on practical implementation issues encountered with regard to resources, data effort, pre-requisite training, knowledge and skills, and complexity of implementation. In addition, outcomes (teacher effectiveness indices) from all methods are compared and discussed to evaluate and understand the consistency of outcomes with regard to teacher ratings based on methodological approach. Finally, the inferential merits of the approaches are discussed, along with their strengths and limitations in broader teacher evaluation systems (Bill & Melinda Gates Foundation, 2011; Gallagher, Rabinowitz, & Yeagley, 2011).

Perspectives

At the close of the 2010 legislative session, the Governor of Arizona signed into law Senate Bill 1040 stipulating that the Arizona State Board of Education (SBE) adopt and maintain a model framework for evaluating classroom teachers using an evaluation instrument that includes quantitative data on student academic progress (Arizona Revised Statutes §15-203 (A)(38)). Specifically, the legislation requires that such quantitative measures account for between thirty-three and fifty percent of a teacher’s overall evaluation score. The legislation goes into effect in 2012-2013.

Methods

For this study, the following analytic approaches were examined: (1) Non-Hierarchical Linear Regression procedures - including and excluding covariates; (2) Hierarchical Linear Models (HLM) - including and excluding covariates; (3) Quantile Regression; and (4) Non-statistical growth measures (i.e., pre-post testing using parallel test forms, tests based on vertically equated scales, and movement of students across pre-defined performance categories). The choice of approaches was based, in part, on the types of analytic growth and teacher effectiveness models being utilized by the Arizona Department of Education (ADOE). In future school years, the ADOE will be incorporating Quantile Regression-based approaches into its accountability systems. In that regard, Arizona has adopted the Student Growth Percentile (SGP; Betebenner, 2011) package as a core analytic environment for measuring student progress and assigning accountability ratings to schools and districts. Additionally, selected districts in Arizona have already begun to define a variety of approaches, both statistical and non-statistical, for quantifying student progress over time.

Results

This study compares and contrasts a variety of methods for measuring teacher value-added from an LEA practitioner’s perspective under the context of meeting state-mandated teacher evaluation requirements. Included will be discussion of implementation resources required, comparison of estimated teacher effects, and suitability to policy and decision-making tasks.

Significance

Arguably, the majority of Arizona school districts are ill-prepared to implement well defined and valid methods for measuring teacher value-added using quantitative achievement data. By investigating, evaluating, and discussing a variety of analytic approaches, it is hoped that LEA assessment practitioners will be better able to implement strategies which suitably serve the intent of accountability legislation, provide instructionally relevant information to teachers and decision makers, and meet the unique conditions and characteristics of the local environment.

Author

• Edward F. Sloat, Arizona State University