Assessing Quasi-Experimental Approaches for Evaluating a College Coaching Program for Students with Foster Care Experience

• In Event: Building and Assessing the Credibility of Causal Inference in Policy Evaluation

Abstract

Introduction/Background: Drawing on the total survey error framework, we use a total causal error framework to compare two quasi-experimental approaches that construct the counterfactual differently.

Purpose/Research Question: When randomization is not feasible and no experimental benchmark is available, how should evaluators compare credible quasi-experimental alternatives? We argue that this comparison should consider the distinct sources of error introduced by different constructions of the counterfactual.

Data: We apply this framework to design an evaluation of Great Expectations, a Virginia college coaching program for students with foster care experience.

Research Design and Methods: One approach uses a within-state student-level comparison group analyzed using comparative interrupted time series (CITS), while the other constructs a comparison group from other states using synthetic difference-in-differences (SDID). We use the total causal error framework to compare the resulting measurement, representation, and identification threats. Their differing error structures also provide a basis for evidence triangulation.

Results/Findings: The two comparison approaches offer complementary strengths. The within-state comparison reduces concerns about cross-state differences in disclosure rules, variable definitions, and administrative processes, while the cross-state comparison reduces the potential for contamination between treated and comparison students. These differing strengths and vulnerabilities create distinct error structures that can inform interpretation across analyses.

Conclusion/Implications: The total causal error framework helps evaluators compare alternative constructions of the counterfactual and identify how their strengths and vulnerabilities differ. Concordance or discordance across approaches with different error structures can then be interpreted through evidence triangulation to strengthen causal understanding.

Authors

• Elizabeth Willow Eisenhauer, Westat
  Presenting Author

• Kevin P Baier, Westat
  Non-Presenting Co-Author

• John Cosgrove, Westat
  Non-Presenting Co-Author

• John D Fluke, University of Colorado Denver
  Non-Presenting Co-Author

• Daifeng Han, Westat
  Non-Presenting Co-Author

• Rachel Mayes Strawn, Virginia's Community Colleges
  Non-Presenting Co-Author

• Kathryn A Henderson, Westat
  Non-Presenting Co-Author