Moderation in Partially Nested Multisite Experiments: Estimation, Inference, and Design-Randomized Trials

• In Event: Design and Analysis of Multisite Experimental Studies to Investigate Treatment Effect Heterogeneity

Abstract

Despite the utility of multisite designs, recent literature has noted that they have not been well-adapted for the increasingly sophisticated nature of contemporary interventions and programs (e.g., Kraemer, 2000; Raudenbush & Bloom, 2015). One type of design that has received growing attention in recent years is experiments with partially nested structures (e.g., Schweig & Pane, 2016; Sterba, 2017). In partially nested designs, the treatment and control conditions maintain different hierarchical structures or forms of nesting. In many intervention studies, for instance, the treatment condition induces a form of nesting or clustering that does not naturally exist in the control condition (e.g., small group tutoring by a tutor introduces a form of dependence or nesting among small group members that does not exist for control students who do not participate in small group tutoring).

In this study, we consider moderation effects within a class of partially nested designs that induce meso- or intermediate-levels in the treatment condition (only) — partially nested multisite (cluster-) randomized trials. The novel feature of this design is that within each site, the intervention induces an additional level of intermediate nesting among people in the treatment condition through participation in a social treatment or experience that does not exist in the control condition. Although a review of prior substantive literature suggests that partially nested multisite clustered-randomized trials are present in theory, the appropriate analysis of moderation within this structure has gone unexamined. To address this gap, we develop the statistical theory that motivates this design, map out its use in practice, and develop estimation methods and closed-form expressions to track moderation effects, their sampling variability and the statistical power to detect those effects. Collectively, the results provide the core tools to effectively and efficiently design and analyze studies drawing on partially nested multisite cluster-randomized design to understand for whom and under what conditions treatments prove effective.

For a brief illustration, consider one type of design within this domain — a three/two partially nested multisite design. Our working example takes up the design of a hypothetical multi-school teacher-randomized design. We examine a study that randomly assigns teachers within school blocks to participate in a control condition or a collaborative lesson study program designed to improve teacher instruction (outcome). Under the control condition, we have a typical two-level hierarchy with teachers nested within schools. Under the intervention condition, teachers are randomly assigned to lesson study groups within schools resulting in a three-level hierarchy for the treatment condition with teachers nested within lesson study groups nested within schools.

To delineate moderation effects within this three/two partially nested multisite structure, we map the design onto a multivariate linear mixed effects model, derive the maximum likelihood estimator of moderation effects and their sampling variability, and develop statistical hypothesis tests and power formulas to support inferences and effectively design studies. The results provide a set of principles and strategies to guide researchers in the design and analysis of moderation in partially nested multisite (cluster) randomized trial.

Authors

• Yanli Xie, Florida State University

• Benjamin Kelcey, University of Cincinnati

• Kyle T. Cox, University of North Carolina - Charlotte

• Fangxing Bai, Montana State University