Using Genetic Matching Methods to Evaluate the Impact of PERC/MSPinNYC2

• In Event: 26.077 - Teens Teaching and Leading: The Peer-Enabled Restructured Classroom in Urban STEM

Abstract

The PERC/MSPinNYC2 program restructures early high school STEM courses to include 6-8 Teaching Assistant Scholars (TAS) who, along with the teachers, facilitate daily in-classroom group work. Early pilot studies suggested the model of peer-enabled restructured classrooms (PERC) increases student achievement and closes the achievement gap in high school STEM courses. The MSPinNYC2 program also attempts to promote TAS college readiness by providing mentoring and a supportive pipeline from high school-to-college. Because the MSPinNYC2 program is a multi-year, multi-site STEM intervention, with new cohorts of students entering the program each year, it is essential to develop a sound, rigorous method for evaluating the effectiveness and the scalability of the intervention—one that can be replicated each year.

The purpose of our study is to evaluate early evidence of program implementation and impact. Specifically, we are seeking early evidence of the effects of MSPinNYC2 on high school students’ achievement in two key STEM disciplines: Integrated Algebra and Living Environment (Biology). Using an evidence-based approach—genetic matching—we ask if, in its early stages, the program is making a difference in students’ academic achievement and college readiness?

Given the non-experimental nature of the program, observational analyses that allow for causal inferences were implemented. Matching techniques can be used to estimate effects of treatments when randomization is not possible allowing one to control for confounders that may have otherwise been controlled for with randomization and experimental manipulations. Specifically, our study utilized Genetic Matching (Diamond & Sekhon, 2013), a multivariate matching method that uses a search algorithm developed to maximize the balance of observed covariates.

The data used for the Genetic Matching procedure included 2 cohorts of PERC students from years 1 and 2 of the program and a large sample of New York City Students from peer schools. After creating the matched sample, regression-adjusted matched estimates were conducted to estimate the average treatment effect on the treated (ATT) examining specifically the effects of the treatment on students NYS Regents test scores.

Results revealed that in years 1 and 2 PERC Integrated Algebra students were not more likely to perform differentially on the Integrated Algebra Regent exam than non-PERC NYC students. Conversely, in both years 1 and 2 PERC Living Environment students were more likely to score higher, pass, and obtain college readiness on the Living Environment Regent exam than non-PERC NYC students. This suggests that there is an immediate effect of the PERC model in the Living Environment course, but not for the Integrated Algebra course.

This evidence of early implementation allows the researchers to understand early on what is and is not working in the PERC model so that it may be further evaluated and adjusted. Furthermore, our study shows the ability to use matching methods as a means of monitoring the efficacy of a large, multi-site instructional intervention like MSPinNYC2.

Authors

• Ally Thomas, University of Pittsburgh

• Sarah M. Bonner, Hunter College - CUNY

• Howard T. Everson, City University of New York