Beating the Odds: How School-Level Factors Influence Success With Urban Advantage

• In Event: Roundtable Session 3
  In Roundtable Session: Understanding the Urban Advantage Middle School Science Initiative’s Implementation and Impact: A Multicity Analysis

Abstract

Our previous impact analyses of the UA-NYC program show that students attending UA schools (schools that are in UA for at least two years), on average, outperform students attending non-UA schools on the eighth-grade science exam (Author, 2012). Results showed that students attending UA schools scored 0.041 standard deviations higher than students at non-UA schools.

This study uses mixed methods to examine the heterogeneity of treatment effects across UA-NYC schools and to examine why these differences may exist. Using random effects models, we identified schools that are “beating the odds” and those that are not performing as well as predicted. Observable characteristics associated with better-than-expected performance include being in UA for four years or more, having a lower percentage of free lunch-eligible students, and having a lower percentage of inexperienced (less than three years) teachers. Thus, even though the random effects model control for school-level characteristics, certain school-level factors are still related to a school’s performance, relative to expectations. And while UA schools perform better than expected compared to non-UA schools, there is still a wide variation in performance among UA schools.

Using a case study methodology, we go further to identify the factors behind UA success by examining UA’s implementation in schools that are beating the odds. Identifying these best practices across different school contexts may help UA program staff develop strategies to help UA schools with more limited success. Additionally, understanding the contexts in which UA is likely to be most successful may provide insight for program staff.

In order to identify best practices, this evaluation dives into the “black box” of UA. We uncover previously unobserved school characteristics and identify how each of the UA strategies works within these schools to improve students’ science achievement. This analysis allows us to examine more nuanced characteristics, such as collaboration among teachers, concentration of UA teachers, and administrative support, that impact successful UA implementation. The results may provide important information about how UA can best be implemented in various settings in order to maximize results. This study is crucial for the successful expansion of UA and development of similar STEM-focused programs in order to allocate funds strategically and improve their effectiveness.

Authors

• Meryle Weinstein, New York University

• Emilyn Ruble Whitesell, New York University

• Michele Leardo, New York University