Partially Nested Multisite Experiments With Repeated Measures: Estimation, Inference, and Design

• In Event: Statistical Power and Optimal Design of Experimental Studies Investigating Main, Moderation, and Mediation Effects

Abstract

Multisite experimental designs are exceptionally common in education for drawing causal inferences about treatment effects (Spybrook, 2014), and arise when we assign students (or teachers) within a school to different intervention programs and replicate this assignment mechanism across multiple schools (Raudenbush & Liu, 2000).

A main interest in educational studies is the growth of the individual within a school. With the purpose of profiling the individual trajectory, multisite experimental designs with repeated measures are commonly used in experimental designs to investigate the course of individual growth, identify critical factors for individuals’ change, and assess the effects of educational interventions and programs. Because multisite studies essentially produce many small, school-specific studies, each school represents an independent experimental study. This type of design allows us to obtain a treatment effect from each school. In addition, the design with repeated measures allows us to explore the growth of individuals within the school context.

However, current experimental designs with repeated measures have not been well adapted for the types of multilevel structures (e.g., Raudenbush & Bryk, 2002). For example, in the study of teachers professional learning program (Benner et al., 2022), teachers assigned to the treatment condition attend the integrated literacy study group to receive professional learning intervention, whereas teachers assigned to the control group are on the waitlist and do not attend a structured professional development program. This type of design gives rise to a partially nested structure. That is, the treatment condition induces a form of nesting or clustering (i.e., observations nested within teachers nested within teacher study groups nested within schools) that does not naturally exist in the control condition (i.e., observations nested within teachers nested within schools).

Current literature either focuses on repeated-measures studies with a fully nested data structure (e.g., Raudenbush, 2001) or partially nested structure (e.g., Baldwin et al., 2011). To fill in the gap, we develop methods for longitudinal studies with partially nested structures. The purpose of our study is to advance experimental design in longitudinal studies by developing principles and strategies for partially nested multisite randomized trials with repeated-measures. Specifically, we formulate explicit models of individual growth, derive estimations of main effects, and develop principles of experimental designs with repeated measures. That is, we begin by outlining the nature of this type of design and detailing the statistical models. We follow with the development of principles of estimation and inferences as well as expressions to track the statistical power. We validate our formulas by simulation studies. Finally, we conclude with the design guidance and directions for future work.

The results provided a set of modeling and estimation principles and formulas to guide researchers in the design and analysis of partially nested multisite clustered randomized trials with repeated measures.

Authors

• Yanli Xie, Florida State University
  Presenting Author

• Benjamin Kelcey, University of Cincinnati
  Non-Presenting Author

• Kyle T. Cox, University of North Carolina - Charlotte
  Non-Presenting Author

• Fangxing Bai, University of Cincinnati
  Non-Presenting Author