Paper Summary
Share...

Direct link:

Impact Study of the National Math and Science Initiative's College Readiness Program: Year 1 Randomized Cluster Trial Results

Fri, April 5, 4:20 to 5:50pm, Fairmont Royal York Hotel, Floor: Mezzanine Level, Confederation 5

Abstract

Objectives: Through a scale-up grant awarded to NMSI by the Investing in Innovation (i3) program, we are conducting an independent evaluation of the impact of the College Readiness Program (CRP) on students’ AP outcomes using a randomized cluster trial with 28 treatment schools and 21 control schools in 10 states. This study focuses on the assessment of the program’s impact on selected student AP exam outcomes as well as AP program participation.

Perspectives: The CRP is an established program designed to promote STEM education in high schools to improve students’ readiness for college. The CRP provides teacher, student, and school supports to promote high school students’ success in Advanced Placement (AP) courses particularly for underrepresented students.

Methods: The impact study research questions focused on the impact of the CRP on the likelihood that student took STEM-related AP courses, as well as the impact of the CRP on the likelihood that students achieved a qualifying score of 3 or higher on STEM-related AP exams, compared to the delayed treatment schools. To address our questions, we conducted a randomized cluster trial (RCT). Schools were randomly assigned to treatment or control conditions and teachers within schools following the school assignment. Schools were randomly assigned to experimental conditions prior to program implementation. In year 1, there were two groups of schools: a treatment and a delayed treatment group which comprised schools in which implementation of CRP involved a one-year delay.

Data: AP test data from a total of 28 treatment schools and 24 delayed treatment schools with 12,366 students and 21 with 8,921 students, respectively, were analyzed for this study. Data on AP courses taken by students as well as AP exam data were based on the enrolled 11th and 12th grade students.

Results: We used a 2-level hierarchical generalized linear model (HGLM) with students nested within schools to estimate the effectiveness of the CRP on the likelihood of a student taking AP courses and attaining a qualifying score of 3 or higher on AP exams. The effectiveness of the CRP was examined in six different subject categories: math, science, English, math or science, math, science, or English, and any AP courses. The HGLM used outcomes in each subject category and a treatment indicator (0 for control; 1 for treatment) as a key predictor in the level-2 model. The estimates were all positive yet not statistically significant. In addition, the results indicated the probability of a student taking at least one AP course or scoring 3 or higher on at least one AP exam is not statistically different between the two groups of schools.

Scientific significance: Findings presented in this paper present the first year of a multiyear evaluation project. Insignificant impact of the CRP might be explained by the fact that students in the treatment schools may not have been exposed to a pipeline of high-quality STEM courses prior to participating in the CRP, and so they may not have had adequate time or opportunity to develop strong conceptual understanding of the material in their earlier coursework.

Authors