Statistical Power for Moderation in Three-Level Multisite Individual Randomized Trials

• In Event: Design and Analysis of Multisite Experimental Studies to Investigate Treatment Effect Heterogeneity

Abstract

Background:
Multisite randomized trials (MRTs), which include multisite individual randomized trials (MIRTs) and multisite cluster randomized trials (MCRTs), have been often used for program evaluation. It is very common to have three-level nested data structure in educational settings, e.g., students nested within teachers/classrooms nested within schools. The three-level MIRTs, in which the individuals within teachers are randomly assigned to the treatment or control condition, enable researchers to investigate the overall effects and heterogeneous effects of a program. The heterogeneous treatment effects may vary across teachers and schools (sites) and may vary depending on the moderator variables. The moderators may be at different levels (e.g., student-, teacher-, and school-levels) and in different scales (e.g., binary or continuous). The (moderated) treatment effects may randomly vary or nonrandomly vary across higher levels. Power analysis is a critical step in designing such studies. Although the statistical tools for power analysis of moderation in two-level MIRTs and three-level MCRTs are available (Dong et al., 2021, 2023), they are not available for three-level MIRTs. In addition, the consequences of ignoring a level of nesting for three-level MIRTs are not clear.

Purpose:
The purpose of this study is threefold: (1) derive formulas to calculate statistical power and minimum detectable effect size difference (MDESD) for moderator effect in three-level MIRTs, (2) validate our formulas and investigate consequences of ignoring one level of nesting using Monte Carlo simulation, and (3) demonstrate power analysis using the software we created.

Methods:
The statistical models for investigating moderator effects in three-level MIRTs may vary depending on three factors: (1) level of the moderator, (2) slope of treatment or moderation across level 2, and (3) slope of treatment or moderation across level 3. Table 1 lists 12 models of combinations of these three factors with more explanation in the notes. For example, Model MRT3-1RR-1 represents the three-level MRTs with treatment at level 1 (i.e., MRITs), with a level-1 moderator, and with random slopes for the moderation/interaction term of treatment and moderator across levels 2 and 3.

By extending Dong et al. (2021, 2023) and Snijders’ (2001, 2005) work, we will derive the formulas for the standard error, MDESD, and statistical power for the moderator effect based on three-level hierarchical linear modeling (HLM) (Raudenbush & Bryk, 2002). We will create Excel-based software. We will conduct simulation to validate the formulas of standard error and power and investigate how power/Type I error changes when one level of nesting (level 2 or 3) is ignored.

Results:
We will present our formulas and simulation results in the final paper. Our preliminary results indicated that the relative bias of our formulas compared with the simulation on standard error and power is less than 3%. Table 2 demonstrates the software module for Model MRT3-1RR-1.

Authors

• Nianbo Dong, University of North Carolina - Chapel Hill

• Benjamin Kelcey, University of Cincinnati

• Jessaca K. Spybrook, Western Michigan University

• Kyle Nickodem, University of Minnesota