Empirical Estimates of Moderator Effects in Cluster Randomized Trials: Implications for Statistical Power and Study Design

• In Event: AERA Roundtable Session Friday 3:45 pm - Gold 4
  In Roundtable Session: RT5: Advances and Implications for Causal Inference and Power in Randomized Treatment Designs

Abstract

This study provides empirical benchmarks for moderator effects in cluster randomized trials (CRTs), using data from five large-scale educational evaluations. We estimated 552 moderator effects at the student, teacher/classroom, and school levels using three-level hierarchical linear models. Results show that moderate to large moderator effects—particularly by race and sex—are possible, challenging the assumption that such effects are uniformly small. However, estimates were less stable for small or imbalanced subgroups. These findings have important implications for power and design. To support future planning, we demonstrate how to apply these benchmarks in PowerUp!-Moderator to conduct power analyses for detecting differential effects in CRTs.

Authors

• Qi Zhang, American Institutes for Research

• Molly E. Cain, American Institutes for Research