Methodogy: Inclusive Regression Discontinuity Designs in Analyses of the Effectiveness of Middle-School Mathematics Intervention

• In Event: Learning Mathematics Through Networked Classroom Activities: 10 Years of Progress?

Abstract

Ten years ago, we were asked about how we anticipated formally characterizing the learning outcomes or forms of student engagement related to the use these environments. At that time, our work was not, as yet, sufficiently rigorous and generalizable. As the papers in the session attest, our progress has exceeded what we might have reasonably anticipated. While some of this work builds on existing approaches, a number of innovations emerging from our work with the capabilities of network-mediated learning and teaching. This paper reports on one of these: Inclusive Regression Discontinuity Design as used to establish the effectiveness of a network-mediated middle school mathematics intervention for a district in central Texas.

Inclusive Regression Discontinuity Design is an extension of traditional regression discontinuity design. Traditional regression discontinuity uses a pre-test, post-test comparison of an experimental group below a cut score on some standardized scale to a group score above the cut score before and after a treatment. If, at the cut score there is a vertical discontinuity, or shift upward, in the regression line for the treatment group relative to the line through the results from the comparison group, this discontinuity is evidence of the success of the intervention (Cook and Shadish, 1994, Shadish, Cook, Campbell, 2001). Typically for education interventions, the treatment is a "pull-out" or self-contained program.

While we will report results from a successful year one mathematics intervention that used network supported activities as part of a pull-out intervention, our general approach to network-supported activity design is "inclusive" in ways that encourage heterogeneous grouping. Consistent with this inclusive stance, in year two of our study the network-based activities were used in classrooms comprised of students with scores both above and below the cut score. This hetrogeneous grouping was considered part of the "treatment". Then, just as with standard regression discontinuity analyses, the pretest-posttest results for students in the treatment with prior outcomes below the cut score could be compared with results from students above the cut score and not part of the intervention. All the conditions for internal validity can still be met with this inclusive regression discontinuity design (IRDD) and we will report our positive findings for students initially scoring below the cut score.
We will report on our similarly positive results for students above a cut score who used highly interactive, network mediated, middle school mathematics activities, in a heterogeneous classroom grouping. Unlike other forms of intervention for underperforming students that might be assumed to be harmful or limiting if also applied to higher performing students, this work formally extends the features of RDD to allow us to show that truly inclusive activity design as supported by next-generation network capabilities can result in significant improvements in outcomes for all students. And while this finding came from work based on using classroom networks, assuming similar internal conditions, IRDD should be able to be extended to a range of more fully socially situated accounts of learning, teaching and activity designs.

Authors

• Walter M. Stroup, The University of Texas - Austin

• Guadalupe Carmona, The University of Texas - Austin

• Vinh Pham, Landmark College

• Celeste Alexander, The University of Texas - Austin