Boston Teacher Residency: Understanding Impact Using Multiple Measures

• In Event: 46.038 - Designing Innovative Urban Residency Programs: Using Multiple Measures to Connect Theory and Practice

Abstract

The Boston Teacher Residency (BTR), founded in 2003 is engaged in a multi-year effort to examine the effectiveness of its graduates and to understand the factors that account for the variation in their performance as teachers of record. Understanding the impact of our graduates has been a complex endeavor due to sample size and data limitations as well as the complexity of the goals the program was created to address. Over the past several years, the program has worked with the Center for Education Policy Research (CEPR) at Harvard University to use value-added modeling to study the relative effectiveness of graduates in terms of producing student learning gains. Because the current testing system in Massachusetts only allows CEPR to estimate value-added scores for approximately 25 percent of its’ graduates, it has taken us several years to build an adequate sample size for conducting the analysis. We received preliminary results from the value added analysis in 2011 (Papay, West, Fullerton, & Kane, 2011), and these results and the discussions surrounding them led to meaningful program changes.
Early on, however, the program recognized that value-added modeling relies on rather narrow measures of student learning. It also treats the classroom as a “black box” and provides little information on the specific teaching practices that graduates are employing, or the type and quality of the intellectual work students in their classrooms are engaging in. In order to understand how we can improve our program and the practice of our graduates, we have expanded our data collection to include the Tripod student surveys to capture student engagement and classroom conditions (Ferguson, 2008). We are also using video-taped observations of classroom practice and coding teaching quality using established rubrics, including those used in the MET Project (e.g., CLASS, PLATO, mQI) (Kane et al., 2012).
We report in this paper the many lessons we’ve learned about the potential and the challenges of collecting, analyzing, and interpreting data on teaching quality to drive program improvement. Our experience suggests that policy makers and leaders of teacher preparation programs need to invest more in building the data systems, and the organizational culture and capacity needed to use such data well.

Authors

• Edward Liu, BPE

• Jesse Solomon, Boston Teacher Residency