Experimental Design and Power for Moderation in Multisite Cluster Randomized Trials to Investigate Treatment Effect Heterogeneity

• In Event: Statistical Power and Optimal Design of Experimental Studies Investigating Main, Moderation, and Mediation Effects

Abstract

The statistical power analysis is now routinely required to demonstrate sufficient power to detect the treatment effects if they exist (e.g., US DoE & NSF, 2013; Kelcey, Spybrook & Dong, 2019). There exist the tools for power analyses of the main effects of multisite randomized trials (MRTs) (e.g., Borenstein & Hedges, 2012; Dong & Maynard, 2013; Konstantopoulos, 2008; Raudenbush et al., 2011) and for power analyses of moderator effects in two- and three-level CRTs (e.g., Dong, Kelcey, & Spybrook, 2018; Dong et al., 2021; Spybrook, Kelcey, & Dong, 2016). Although Dong, Kelcey, & Spybrook (2021) and Raudenbush & Liu (2000) provided a framework for power analysis of moderator effects in two-level multisite individual randomized trials, and Bloom and Spybrook (2017) developed formulas for the minimum detectable effect size difference (MDESD) for the site-level binary moderator in MRTs, there is no comprehensive statistical tool for power analyses of moderator effects in three-level MCRTs.

To address this gap, the purpose of this study is to develop a statistical and empirical framework for designing three-level MCRTs to investigate the moderated treatment effect. Specifically, we will derive formulas for calculating the statistical power and the minimum detectable effect size difference (MDESD) with confidence intervals for investigating the effects of various potential moderators in three-level MCRTs.

Conceptual and Statistical Framework

In designing moderation studies in three-level MCRTs, there are multiple considerations concerning the potential form of the moderation including (a) level(s) of the moderator variables, (b) the random or nonrandomly varying slopes (coefficients) of the treatment and moderator variables, and (c) the scale of the moderator(s) (e.g., continuous or categorical). Figure 1 presents the conceptual framework. Table 1 presents the list of moderation designs in three-level MCRTs. We develop the formulas for the standard error (SE), statistical power, and the MDESD and its CIs for the moderator effect at level 1, followed by levels 2 and 3. The formulas are presented in Table 2.

Monte Carlo Simulations

We conducted Monte Carlo simulations to examine whether the standard error and power formulas that we derived were consistent with the simulated results. Tables 3 and 4 present the results of standard error (SE) and power (or Type I error rate) estimates from the Monte Carlo simulations that were calculated based on the formulas using the same design parameters, and the coverage rate of 95% CI for random slope and nonrandomly varying slope models, respectively. The results provided evidence of the close correspondence on standard errors and power (or Type I error) between our formulas and the empirical distribution from the simulation.

Demonstration

We demonstrate a power analysis of moderation in MCRTs using the software that we developed. The results are presented in Tables 5-7 and Figure 2.

Conclusion

This study provided a conceptual and statistical framework to guide the design and analysis of MCRTs. These results will have the potential to substantively impact our understanding of treatment effect variation by providing comprehensive tools to researchers to design rigorous multisite moderation studies.

Authors

• Nianbo Dong, University of North Carolina - Chapel Hill
  Presenting Author

• Benjamin Kelcey, University of Cincinnati
  Non-Presenting Author

• Jessaca K. Spybrook, Western Michigan University
  Non-Presenting Author