Applications of Synthetic Control Models in Educational Research

• In Event: The New Kids on the Methods Block: Using New Methods in Existing Data to Get New Answers

Abstract

Following a large growth in educational data collection over the last several decades, there has been increased interest among researchers in education and developmental psychology in applying time series methodologies commonly used in economics to measure the causal impact of various policies and interventions (Hallberg et al., 2018). These methodologies, which require several time points of observations for units on a given outcome, permit causal inferences from non-experimental data, and have been found to resemble results from randomized control trials (Hallberg et al., 2018). Yet, education data using aggregate measures frequently fail to meet the assumptions necessary to make valid causal inferences from such methodologies. For example, estimates from difference-in-differences models – commonly used to estimate the impact of educational policies – will be biased when pre-treatment trends on the outcome of interest for the treatment and comparison groups are not parallel. Attempts to control for differences in pre-treatment trends using a comparative interrupted time series (CITS) design will similarly be biased if the observed trend differences are not sustainable in the post-treatment period.

To resolve the inherent weaknesses in these methodologies, I present the synthetic control method (SCM) – a relatively recent method derived from the economics literature – as an alternative and more valid approach for such contexts. SCM synthesizes a control unit from a weighted sum of donor comparison units by matching outcomes and explanatory variables in the pretreatment period of the donor units to the same variables in the pretreatment period of the treated unit (McClelland & Gault, 2017). A well-fit synthetic control unit will appropriately match both the intercept and slope of the outcome measure in the pretreatment period. Placebo tests for statistical significance bypass the need for larger sample sizes that are typically necessary to avoid an underpowered analysis. Recent developments in SCM allow for the estimation of treatment effects for multiple treated units, thereby increasing its application across various contexts in education and developmental psychology.

In this paper, I use a real-world example to demonstrate the intuition behind adapting a model from difference-in-differences, to CITS, and finally to the SCM. The example is derived from a replication study examining the impact of integrated student support intervention on student achievement. Previous studies of the intervention in a single school district have demonstrated significant positive impacts on various student outcomes, but the degree to which the intervention impact replicates in additional sites has not yet been analyzed (Walsh et al., 2014). Specifically, I examine the impact of the intervention on English Language Arts (ELA) achievement for 3rd – 5th graders in five low-performing schools in a medium-sized urban and high-poverty district.

Author

• Noman Khanani, Boston College
  Presenting Author