Understanding the Counterfactual for Program Evaluation of Career and Technical Education

• In Event: Measures, Data, and Counterfactuals: Approaches to Improving Research in Career and Technical Education

Abstract

Career and technical education (CTE) encompasses a broad array of educational programs intended to provide students with specialized skills in trades, applied sciences, and modern technologies directly preparing these students for specific careers. Interpreting the results of CTE program evaluations can be challenging given the broad opportunities available for selection into and out of the largely voluntary programs, the variety of contexts in which CTE is delivered, and the multiple avenues available to students for obtaining CTE experiences. While numerous studies have attempted to evaluate the impact of CTE programs, see Rosen, Visher, and Beal (2018) for a review, few studies provide convincing quasi-experimental or experimental evidence on the causal effect of such programs on student outcomes.
This paper examines the problem of understanding the counterfactual population that forms the basis of program evaluations of CTE by taking a case study approach through the lens of typical methods used to study CTE program effects. The paper begins by discussing the key identification strategies and the associated assumptions that have been used to identify program effects including regression and propensity score matching, instrumental variables, regression discontinuity designs, randomized control trials, and lottery-based admissions. The paper then presents a series of case studies evaluating the counterfactual for specific studies that used each of these identification strategies.
The first case study looks at regression and matching methods by examining an evaluation of the Linked Learning Initiative (LLI) in California. Through this initiative, school districts were systematically selected for the development of certified CTE pathways and students were free to select into or out of those programs. As a result, this evaluation could only control for student attributes captured in traditional administrative data. The second case study examines an evaluation by Cullen, Jacob, and Levitt (2005) of stand-alone career academies in Chicago Public Schools (CPS). This evaluation identified the effect of attending an academy by instrumenting for attendance using physical proximity between each student’s residence and the closest career academies. The third case study discusses an evaluation of the Connecticut Technical High School System by Brunner, Dougherty, and Ross (2019) who used a regression discontinuity approach exploiting the fact that students are admitted in order based on application scores. The final case study looks at randomized control trials and lottery-based admissions with a focus on MDRC’s evaluation of career academies housed within nine individual high schools located across the country (Kemple 1997; Kemple and Clayton-Scott 2004; Kemple and Willner 2008).
The questions of each study’s internal and external validity as well as how each study addresses the counterfactual experiences of the students in the control group are explored in depth. In many cases, studies with strong internal validity provide evidence that can be harder to generalize than studies documenting outcomes from broader populations participating in large, long-standing programs. The paper thus concludes by discussing the strengths and weaknesses on these dimensions of each study design type as well as the specific cases presented in this work.

Authors

• Rachel Rosen, University of Michigan

• Stephen L. Ross, University of Connecticut

• Eric J. Brunner, University of Connecticut