Examining Specific Group Effects Through Differential Effects Meta-Analysis

• In Event: Methodological Approaches for Equity-Oriented Evidence Synthesis

Abstract

Effect heterogeneity and subgroup effects - which we refer to as “specific group effects” - are pivotal yet often overlooked aspects of educational research, constraining the development of comprehensive and precise pedagogical interventions. Average treatment effects, while insightful, may obscure heterogeneities across specific groups. The assumption that an intervention has a uniform impact across diverse settings or cohorts is overly simplistic, potentially misrepresenting the reality of its effects. To optimize the efficacy of educational strategies and policy implementation, understanding effect heterogeneity - the variation in outcomes across different contexts, population subsets, and temporal scales - is vital.

Analyzing specific group effects allows us to dissect this variation. It helps to uncover differential impacts of educational strategies and policies, paving the way for more tailored interventions. By highlighting the nuances in the effectiveness of strategies across diverse subgroups, we can ensure educational equity and optimization. Investigating whether an intervention works differently across races or socioeconomic statuses, for example, can allow for more nuanced conclusions and targeted interventions.

By re-analyzing effectiveness data with a focus on differential effects, we can generate detailed effect estimates for each specific group before synthesizing them into a unified meta-analytic model. To do so, we have collected eight primary datasets - all examining the effects of a K-12 school-based SEL program and measuring SEL outcomes - and have begun to estimate specific group effects for each. We plan to ask for datasets or specific group summary statistics for every US-based study included in Cipriano and colleagues’ (2023) recent meta-analysis. We estimate that approximately 15% of the 140 US-based studies will respond to our request and provide summary data. If so, 21 studies will provide specific group data.

Once we’ve re-estimated the effects, we will estimate a correlated and hierarchical effects meta-analytic model, which account for both within-study and between-study variances, and random-effects models, where the true effect can vary across studies. Our dataset has correlated effects because the same construct - SEL skills - will be measured repeatedly on the same participants. Our dataset also includes hierarchical effects - the specific group effects - where one study collects information across various groups of people. Pustejovsky and Tipton’s (2022) correlated and hierarchical effects meta-analytic model allows both sets of effects to be estimated simultaneously. Following best practices for investigating effect heterogeneity, we will examine moderators that may explain why effects differ across groups using a series of meta-regression models.

Investigating effect heterogeneity and specific group effects will provide a richer, more nuanced understanding of educational processes and outcomes. We anticipate that the results of this re-analysis, using a differential effects meta-analysis, will help us provide greater context to SEL programming and inform future practitioners seeking to implement these programs.

Author

• Joshua R. Polanin, American Institutes for Research