Paper Summary
Share...

Direct link:

Measuring Implementation of a STEM-Focused College Readiness Program

Fri, April 5, 4:20 to 5:50pm, Fairmont Royal York Hotel, Floor: Mezzanine Level, Confederation 5

Abstract

Objectives: This study represents first year findings from a three-year evaluation study measuring implementation of the College Readiness Program (CRP) designed to improve the quality of STEM and ELA instruction, expand AP course offerings, and increase the number of students taking and passing AP exams. This study focuses on determining the extent to which key components of the CRP were implemented as intended and identifying facilitators and barriers to implementation.

Perspectives: The CRP was created to raise the academic bar in public schools by demonstrating that more students, especially high-need students, can master rigorous AP coursework. Three factors enable the CRP to address challenges faced by high-need students: Focus on high school reform— CRP schools rethink their AP culture by adopting open enrollment and recruiting more students, including high-need and traditionally underrepresented students; Investment in building the pipeline of students prepared for rigorous coursework; Program structure designed to work within the framework of existing schools.

Methods: The fidelity matrix defines key components of the program, measures of each, scoring rubrics, and fidelity criteria. We measured fidelity separately for each program component (school, teacher, and student supports), defining threshold values to determine the extent to which the program model was faithfully reproduced at each site. Elements were considered to have been implemented with fidelity if in at least 80% of the schools they were implemented as planned or if 80% of targeted staff or students satisfied a requirement.

Data Sources: The research objectives were to better understand teacher and administrator perspectives on the effectiveness or impact of the CRP on student interest and success in AP and school culture. In year 1, data were collected from 129 teachers, and 16 administrators from 28 treatment schools in 10 states as a part of randomized cluster trial. Implementation information was collected from administrative records, surveys, and interviews.

Results: Data reported on here are for the year 1 teachers (129: 78% of the eligible sample). Seven specific school support measures were identified, and in each case the program was implemented with fidelity. Eight specific metrics were identified to evaluate teacher participation and NMSI support. In the aggregate, seven program elements were implemented with fidelity. Out of a maximum aggregate score of two for the student measures, 12 schools scored the maximum (43%) and all others scored one. Data indicated not all program elements were implemented with high fidelity; however, overall results indicated 23 schools (82.1%) achieved 80% or better implementation fidelity, for an average fidelity score of 89.3%.

Scholarly Significance of the Study: Data from the study provide both formative feedback on program implementation as well as information on factors which may be associated with program success. Teacher and administrator perspectives provide unique insight and valuable support for implementation data we are gathering for the evaluation study. Perspectives of those who are implementing the CRP helps determine which factors are most important in creating and sustaining an accessible and successful AP program.

Authors