Observing Special Education Teachers in High-Stakes Teacher Evaluation Systems

• In Event: District and State-Level Approaches to Teacher Improvement

Abstract

All 50 states now require classroom observation systems in teachers’ evaluation systems, and there is evidence that schools are using the information derived from observations – more than from student learning outcomes – to foster teacher development and to guide human capital decisions (Author, 2015c). However, no research studies have examined the validity of general observation systems for use with special educators, a group that represents approximately 14.5% of the teaching workforce. Further, researchers have raised concerns about whether such systems appropriately account for the kinds of instructional practices most commonly used by special educators (Author, 2015b). To help identify effective special education teachers and to help those who are struggling struggling to improve, it will be critical to districts that they have accurate, reliable observation data.

Theoretical Framework

In the present study, we investigate the extent to which the Framework for Teaching (FFT), a widely-adopted observation scheme, reliably and validly captures special educators’ practice. To do so, we adopt Kane’s validity argument approach, in which we appraise the plausibility of the argument that judgments of special educator teaching quality can be made on the basis of FFT scores. This argument is to be validated through an empirical evaluation of four related sets of inferences: 1) scoring, 2) generalization, 3) extrapolation, and 4) interpretation.

Methods and Data Sources

In conducting our validation, we collected classroom observation data from 80 special education teachers in Rhode Island and Idaho. We included elementary and middle school teachers in grades 3-8. Each classroom teacher was videotaped four times in 2016-2017; teachers also completed surveys and were interviewed about their backgrounds, teaching context, and evaluation experiences. All 320 teacher lessons were scored on both FFT and on a set of observation instruments more closely aligned with special education teaching practices, the Classroom Observation Student-Teacher Interactions (COSTI) and the Quality of Classroom Instruction (QCI) (Doabler et al., 2014; 2015). Scores on each protocol are also compared to teachers’ value-added scores.

Results

In the proposed presentation, we will discuss two sets of results from our validation efforts. First, we will present psychometric evidence on the use of FFT with special educators, including: a) do the FFT dimensions conform to theory, b) what are the sources of variance in FFT scores (e.g., rater, lesson, teacher), and c) to what extent do FFT scores converge with other sources of evidence of special education teaching quality, including COSTI/QCI scores and student performance data. Second, we will present descriptive findings from the teacher interviews and surveys to provide data on special educators’ perceptions of the FFT, specifically its usefulness for supporting their development.

Scholarly Significance

There are complex conceptual and logistic challenges in adopting a common evaluation tool across all teachers in a state. While our results examine the case of special educators, it is easy to imagine other populations of teachers for whom similar challenges might emerge. In the absence of empirical data on the FFT’s validity with various subgroups of teachers, it will be difficult to know whether the instrument can help improve teachers’ instruction.

Authors

• Nathan Jones, Boston University

• Courtney A. Bell, ETS