The Impact of Programs of Study on Student Achievement: Interim Results from a Longitudinal Study

• In Event: Programs of Study as Locally Adaptable High School Reforms

Abstract

Objective
This longitudinal study, currently in its third of four years, estimates the impact of programs of study (POS) on student academic and technical achievement. This paper examines the impact of POS on students’ grades and test scores in two urban districts contrasted to carefully matched controls.
Framework
Although the largest effects of POS should be most obvious after a full four years of study in one career-focused area, interim data from multiple measures of achievement should trend toward the POS students.
Sample
Multiple POS-based Career Academies in two districts are participating in the study. In West District the schools are over-subscribed and students are admitted through a randomizing lottery. For the purposes of studying effects of the programs, this creates an RCT (Shadish, Cook, & Campbell, 2002). Admission to the POS in East District follows a more complex procedure. The LEA’s research unit propensity matched a control sample of 9th graders, creating a quasi-experimental design.
Method
POS students’ achievements are compared to the control groups on academic GPA and tenth-grade test scores. We used analysis of covariance (ANCOVA) for academic GPAs and chi-square for test scores. The study makes extensive use of mixed methods (Teddlie & Tashakkori, 2009). Given space limitations the results presented in this proposal are quantitative, we will site supportive qualitative data in the full AERA paper.
Results
Academic GPAs were calculated using math, science, English, and social studies grades over the first two years of high school. Controlling for 8th grade test scores, students randomly assigned to attending West district’s POS scored significantly higher on their state’s 10th grade tests than did their matched peers. The effects held in reading (Cohen’s d = .10, p <.01), math (d= .17, p<.01), and science (d= .10, p<.05). West’s students’ Academic GPAs favored the POS students (exp. = 2.73, cont. = 2.65, d =.11), however the result was not statistically significant.
In the East district’s comparisons, end of course exam scores in Algebra 1 favored the experimental group (d = .15, p<.01), were not-significant in Geometry and Biology, but favored the control group in Algebra 2 (d = .18, p<.01). There were no significant differences between POS and control students on unweighted GPA, but when grades were weighted to reflect more rigorous courses, results favored the POS group (POS weighted GPA = 2.71, control = 2.55, d = .15, p<.01).
Discussion and Conclusions
The final two years of high school are when POS theory predicts students most likely to excel as POS classes become a larger portion of the treatment students’ school day. Yet after only two years, differences in GPAs and standardized tests largely favored POS students. These results may help policymakers decide the best ways to improve education to help all students succeed.
Implications for the remaining years of this study and for POS more generally will be discussed.

Authors

• Marisa E. Castellano, University of Louisville

• Kirsten Ewart Sundell, University of Louisville

• Laura Overman, University of Louisville

• Oscar A. Aliaga, University of Louisville