The Impact of Comprehensive Support and Improvement Designation on Student Outcomes

• In Event: Sheraton Roundtable Session Three
  In Roundtable Session: Improving Low-Performing Schools Through Accountability Policy Under the Every Student Succeeds Act (Table 2)

Abstract

Several recent meta-analyses demonstrate substantial heterogeneity in the effects of accountability on student outcomes (e.g., Schueler et al., 2020). Given the flexibility under ESSA regarding identification and supporting CSI schools, we might expect substantial heterogeneity in effects across states. Having better understanding of where accountability has spurred improvement and where it has not provides an opportunity to investigate why accountability is successful in some contexts and not others and whether certain design elements or approaches to supporting low-performing schools are more successful than others.

Methods – To investigate the effects of the CSI designation on student outcomes, we obtained student- and school-level data from three states. Outcome data in each state consist of student-level test scores in math and ELA, attendance rates, student discipline, and graduation. The three states also provided student-level demographic data including student race, economic disadvantage status, special education status, and English learner status. We paired student-level data with data on CSI designation in 2018-19 and subsequent years.

Using this data, we used two approaches to examine the effects of CSI designation on student outcomes. First, we used regression discontinuity to investigate the effect of CSI designation around school performance cutoffs. Ohio and Florida both use an aggregate school performance index where multiple school performance measures are combined into a single school value representing overall school performance. Schools performing below a cutoff are designated as CSI schools. In California, a series of cutoffs on each individual indicator are used to determine performance ratings on each indicator. Schools with low ratings on multiple indicators are then designated as CSI schools. Given that no single cutoff determines CSI status, we used a comparative interrupted time series approach for California, that establishes performance trends in both CSI and non-CSI schools prior to initial designation and then compares the difference in trends for both school types in the post-identification time periods.

Results – Results show relatively few statistically significant differences during the first year of implementation. Exhibit 3 illustrates this overarching conclusion and presents the regression discontinuity estimates of the effects of CSI designation on student outcomes in Florida in the first year. Given that school improvement often takes time, with the first year of implementation often being a planning year, insignificant findings are not altogether surprising. Soon, we will collect outcome data through the 2021-22 school year. We hope to have additional results for more recent school years to share in time for the AERA conference in Spring 2023.

Significance – The goal of school accountability policies is to spur improvement. In particular, the policy of identifying CSI schools and providing supports and resources to those schools is intended to bolster student outcomes in schools that have been identified as the lowest-performing in the states. This study shows whether the policy has been successful in achieving that goal. When paired with findings from the principal survey and district interviews, we will start the process of understanding whether certain accountability policy features potentially promote or hinder the success of accountability.

Authors

• Umut Ozek, RAND Corporation
  Presenting Author

• Drew Atchison, American Institutes for Research
  Non-Presenting Author

• Katie Hyland
  Non-Presenting Author

• Damon Blair, American Institutes for Research
  Non-Presenting Author