Understanding CTE Program Impacts and How and Why They Vary

• In Event: New Evidence on the Effectiveness of Career and Technical Education

Abstract

This paper will provide unique insights into the conditions under which CTE is most and least effective and for whom. Though this research is situated in the study of CTE in New York City, our strategy for developing measures of CTE program characteristics and methods for assessing impact variation can be extent to a wide range of contexts.

This paper answers the following questions:
1) To what extent do CTE program features explain variation in school-level impact estimates of CTE programs?
2) To what extent do CTE programs have an impact on disparities in college and career readiness across student subgroups identified by prior education experiences as well as gender, race/ethnicity, English learning needs, and socio-economic status?

This research is situated in NYC, the largest public school district in the United States. The district uses a high school selection process that requires students to list up to 12 programs for admission. Of the approximately 700 available high school programs, 240 are CTE programs and about 60% of them have been approved by the state. Programs seeking state approval must submit a program approval form (PAF) that provides details about its teachers’ certifications, work-based learning opportunities, professional certifications, and school partnerships with colleges and employers. Thus, the PAFs provide data on how program components differ across NYC CTE offerings.

We first construct the potential moderators of impact. We use existing theories of change about CTE efficacy and impact to inform principal components analysis (PCA) on the PAF data as a form of exploratory factor analysis. We then use the results of the PCA to create composite measures of program characteristics to describe differences by program. For the moderator analysis, we conduct site-level impact analyses using propensity score matching (PSM) methods that have been validated against naturally occurring randomized control trials (RCTs) within our full sample. We test pre-specified hypotheses by estimating differences in impacts across subgroups of sites reflecting variation in the composite measures. Finally, we replicate these analyses to test for systematic differences in impacts for key student subgroups.

Data on student characteristics and outcomes are from administrative records available through our partnership with the New York City Department of Education. The data include more than 350,000 students who entered high school between 2013 and 2017. We can follow these students through high school and into college through the 2019-2020 school year. The analysis sample consists of nearly 30,000 students who were assigned to high school program offering CTE between 2013 and 2017. The matched comparison group consists of a similar number of students who were not assigned to CTE programs during the same period.

Our analyses are ongoing. We find that our PSM approach accurately replicates both baseline equivalence and impact estimates generated by naturally occurring CTE admission lotteries (RCTs) for nearly 8,000 students in our sample. The extension of the PSM approach produced baseline equivalence meeting What Works Clearinghouse standards.

Authors

• Rebecca Unterman

• James J. Kemple, New York University

• Shaun Michael Dougherty, Vanderbilt University

• John Sludden, New York University

• William Spagnola, New York University

• Alec Gilfillan, MDRC