Power Analysis for Main and Moderator Effects in Multisite Longitudinal Experiments With Site Fixed Effects

• In Event: Design and Analysis of Multisite Experimental Studies to Investigate Treatment Effect Heterogeneity

Abstract

Multisite longitudinal experimental designs have been frequently used in educational interventions, where for example, students from the same schools are randomly assigned to a treatment or control condition and then are followed over time. One purpose of longitudinal studies is to explore how the treatment effects change over time. Also, educational researchers are interested in evaluating whether the change of the treatment effects varies among subgroups of students or schools. These student and school characteristics are usually called moderators, and the moderator effects could be examined through interaction terms between a treatment and student or school characteristics within regression analyses.

Considering the typical nested data structure in multisite longitudinal studies (e.g., repeated measures nested within students nested within schools), multilevel models (MLM) with site random effects have been traditionally used to estimate the treatment effects on the linear or nonlinear rates of change (e.g., Raudenbush & Liu, 2001). It should be noted that these models assume the treatment indicator is independent of the site random effect to estimate the casual treatment effects on the linear or nonlinear rate of change (referred to below as the main effects).

Recent discussions on the design and analysis of multisite experimental studies (e.g., Miratrix, Weiss, & Henderson, 2021) suggested setting site effects fixed (e.g., site dummy variables) because of the concerns about potential correlations between treatment indicator and the site random effect. For example, the demand for a particular treatment might be higher for some schools than others, in which case the treatment is correlated with site characteristics represented by the site random effects in MLMs (Miratix et al., 2021).

One critical consideration when designing longitudinal cluster experiments is to decide the sample size allocation across levels and treatment conditions to guarantee adequate power to detect the effect of interest (Cohen, 1988). Prior studies have developed statistical power computation formulas to detect the main and moderator effects for three-level longitudinal multisite experiments using MLMs with random site effects (e.g., Li & Konstantopuolos, 2019, 2021). However, similar methods for the models with fixed site effects are lacking.

This study contributes to the literature by providing the statistical power analysis methods and tools for longitudinal three-level multisite experiments (e.g., repeated measures, students, and schools), where the third-level units (sites) are treated as fixed effects (i.e., site dummy variables). We consider both the main and level-2 moderator (e.g., student gender, race, etc.) effects. Note that we cannot include a level-3 moderator (e.g., school enrollment) in the model because it will be perfectly colinear with the site dummy variables. Our methods consider both binary and continuous moderators, whose effects could be fixed or non-randomly varying among sites. We will implement the methods developed from this study into an R package and a Shiny App to help applied researchers plan longitudinal experiments. Illustrative examples will be provided to demonstrate the applicability of the methods and tools in power computations.

Authors

• Wei Li, University of Florida

• Spyros Konstantopoulos, Michigan State University