A Multilevel Propensity Score Approach to Accelerate Learning to Improve in a Networked Improvement Community

• In Event: 58.038 - Taking the "Measure" of Networked Improvement Communities (NICs): Examining the Role of Measurement Across a NIC

Abstract

Propensity score matching is a statistical technique used in observational or nonrandomized studies to estimate the effect of a treatment or intervention by accounting for characteristics that influence treatment selection (Rosenbaum & Rubin, 1983). For instance, certain kinds of students may have been more likely to enroll in an educational intervention program, leading to more positive outcomes than there otherwise would have been. This matching method attempts to reduce such selection bias and increase the validity of causal inference. More recent applications of this method take a multilevel modeling approach to account for the hierarchical nature of educational data (Hong & Raudenbush, 2005, 2006). Although propensity score matching (and its multilevel version) is typically used for evaluation research, the objective of our study is not only to assess the effectiveness of the Pathways programs, Statway and Quantway, but also to illuminate areas of improvement in our interventions to inform learning and decision-making.

To construct a matched comparison group, our analytic procedure is applied to each Pathway as follows: (a) formulate a two-level model (students within colleges) with a set of covariates including student background characteristics and prior course attempt/success patterns to obtain propensity scores; (b) conduct matching per college with a nearest neighbor matching algorithm (Rosenbaum & Rubin, 1985), which is appropriate for our study because we want to retain all Pathways students and have a large pool of non-Pathways students available for creating matches; and (c) estimate an effect of each Pathway by comparing course success rates of the Pathways students against those of their matched counterparts by using a four-level model. In this four-level framework, matched clustered students (level 1) are nested within the Pathways students (level 2), who are nested within the Pathways faculty (level 3), who are then nested within their colleges (level 4). Because matched comparisons are formulated for each Pathways student, their respective comparison students are also assigned the corresponding Pathways faculty ID. This strategy allows us to form each faculty member’s classroom as a mini-experiment in which the mean outcome of their Pathways students can be compared with that of similar students who pursue traditional math courses, and hence, to estimate the variability in effect among faculty within colleges.

We will present multilevel modeling results from both Pathways, separately, to demonstrate their overall effectiveness and variation in effect at the college and faculty levels. More importantly, we also discuss ways in which we can leverage our findings to facilitate network-wide improvement. Specifically, we present data visualizations that plot college and faculty effect sizes against reference data in order to identify positive deviant colleges and faculty members. This brand of learning is essential, because Networked Improvement Community (NIC) members implement the Pathways across a diverse range of contexts, such as differing institutional racial/ethnic compositions and the number of academic terms through which Pathways courses are taught. By investigating how positive deviants effectively adapted the Pathways to local contexts, we may share our learnings with other NIC members to accelerate learning to improve the Pathways.

Authors

• Hiroyuki Yamada, Carnegie Foundation

• Melrose Huang, The Carnegie Foundation for the Advancement of Teaching