Comparison School Selection for Comparative Interrupted Time Series Analysis

• In Event: 21.021 - Implementation Evaluation of a High School College Readiness Program: Design, Methodology, and Findings

Abstract

Objectives: In addition to the randomized cluster trial (RCT), this study conducts a quasi-experiment for comparing effects of the College Readiness Program (CRP) between the study schools (28 treatment and 21 delayed treatment schools) in 10 states and a set of schools otherwise similar to the study schools. This paper focuses on selection of the comparison schools using propensity score methods, and lays out comparative interrupted-time-series (CITS) methods for the estimation of the effects of the CRP on student AP outcomes.

Perspectives: Despite the lack of randomization, the quasi-experiment takes advantage of the fact that more years of data prior to the CRP intervention are available, allowing one to employ a longitudinal perspective. In contrast to RCTT estimates, CITS analysis provides a way of examining a pattern of enrollment/performance over multiple pre-intervention years, and allows for more reliable estimation of the baseline as well as the boost in the enrollment/performance.

Methods: To obtain rigorous intervention effects that approximate causal estimates, we used propensity score methods in selecting comparison schools. Propensity score is a scalar value that summarizes the likelihood for a unit to receive a treatment, often based on a large set of variables. The specification of propensity score models included covariates, student AP course enrollment and performance, and school background characteristics in the pre-intervention years. We used a propensity score strategy based on matching and stratification that selects one comparison school per study school within the same state.

Data: For school demographic characteristics, Common Core of Data from the National Center for Education Statistics were used. For student AP course enrollment and performance, AP exam data were obtained from the College Board.

Results: We first identified districts that have a sufficient number of high schools that are similar to CRP schools in demographics and in AP course enrollment and performance. This identified a pool of 405 schools in multiple districts in the 10 states. The matching strategy selected 49 comparison schools, where most bias arising from all observed covariates was eliminated. The standardized mean difference in linear propensity scores between the study schools and the pool of 405 schools was 1.05. In the matched sample, this bias estimate was reduced to 0.10; there was also significant improvement in the balance in most individual covariates, such as AP enrollment size, percent free or reduced lunch eligible students, and school type and regions.

Scientific significance: Although experiments are a gold standard in reaching more clear-cut conclusions, in practice even experiments can be affected by difficult issues: generalizability of the study sample, heterogeneity of intervention effects across sites, contextual differences both in the pre-intervention and intervention periods, etc. By providing alternative designs based on matched sample, and alternative yet reasonable estimates of intervention effects (using CITS based on the matched sample) within the same umbrella of the study, the study can provide ample opportunity to better understand why, for whom, and under which settings the CRP tends to be more effective.

Authors

• Jinok Kim, University of California - Los Angeles

• Kilchan Choi, University of California - Los Angeles

• Richard S. Brown, West Coast Analytics

• Eva L. Baker, University of California - Los Angeles