Design and Analytic Features for Reducing Biases in Skill-Building Intervention Impact Forecasts

• In Event: Regression Toward the Mean, Lord’s Paradox, and Modeling or Forecasting Change in Longitudinal Analyses

Abstract

The longer-term impacts of educational interventions are highly policy-relevant, but they are relatively rare. A common solution to this problem has been to rely on longitudinal correlational research to determine the most optimal targets for intervention. However, in the case of academic achievement, this approach has sometimes yielded overly optimistic predictions about the medium-term effects of successfully improving early skills. We test a set of approaches for more accurately forecasting the medium-term impacts of skill-building interventions using data from the Number Knowledge Tutoring program that followed 639 students from 40 schools and 227 classrooms from a southeastern metropolitan district from first to third grade (Fuchs et al., 2013). Students at-risk of low academic performance were randomly assigned to either a control group, tutoring with speeded practice, or tutoring with non-speeded practice during first-grade. We implement a within-study design (Cook, Shadish, and Wong, 2008) to compare the average forecast of increasing first grade math skills on third grade math skills calculated from experimental estimates from the treatment groups and correlational estimates from the control group.
The forecasts are estimated using the product from the estimated treatment impact on aa post-treatment outcome in first grade (using the full sample) and the estimated effect of the post-treatment outcome on the medium-term outcome, measured 2 years later (using the control group only). We assess forecast accuracy to the intervention groups with three different analytical approaches: forecasting using a single post-treatment outcome (Figure 1a), assuming each post-treatment outcome has an independent causal impact on a medium-term outcome (Figure 1b), and assuming that post-treatment outcomes share causal impacts on a medium-term outcome (Figure 1c).
We estimate regression paths bi in numerous ways to model potential bias introduced from variations of study designs and analytical approaches such as the extent to which the covariates included measurements aligned specifically to the intervention. Results are shown in Figure 2, where forecasted effects (y-axis) are plotted against the observed experimental impacts (x-axis) such that the most accurate forecasts would fall along the diagonal line. We modeled the direction and magnitude of omitted variables bias finding that demographic variables that are correlated to the outcomes and pretests of the skills measured are necessary covariates but not sufficient to produce unbiased forecasts in all cases (Figure 2 Panel A). We also find that of the three approaches shown (Figure 2 Panel A-C) forecasting with a single post-treatment outcome yielded the most accurate forecast on average. Finally, the accuracy of this forecast improved when the forecast was calculated using both a measure of early math skills that was closely aligned with the content taught during the intervention and a more comprehensive measure of math achievement.
Improving the accuracy of our forecasts of the medium-term impacts using observed post-treatment impacts could lead to more efficient design and investment in educational interventions, enhance policy decisions, inform statistical power calculations for intervention evaluation, and provide riskier tests to corroborate theorized causal processes (Waller & Meehl, 2002).

Authors

• Daniela Alvarez-Vargas, University of California - Irvine
  Presenting Author

• Sirui Wan, University of Wisconsin - Madison
  Non-Presenting Author

• Drew Bailey, University of California - Irvine
  Non-Presenting Author