Design Parameters for Planning Multilevel Randomized Trials on Social and Behavioral Outcomes

• In Event: Improving the Design of Program Evaluations in Education and Prevention Science

Abstract

Multilevel randomized trials are now widely used to examine intervention effects in prevention science (National Research Council & Institute of Medicine, 2009). In designing multilevel randomized experiments with sufficient statistical power to detect a meaningful effect size of an intervention with precision, researchers need to make reasonable assumptions about the design parameters to estimate the required sample sizes. Three critical design parameters associated with statistical power are: (1) effect sizes, (2) intra-class correlations (ICCs), and (3) proportions of variance explained by a covariate at different levels (R-squared). Recently, several studies have been conducted to report the ICCs and R-squared for academic achievement outcome measures and outcome measures for teacher professional development. However, there is limited information about these design parameters on social behavioral outcomes (Dong et al, 2016).
In addition, it is meaningful to use empirical benchmarks for interpreting effect size in prevention science. More recently, Hill, Bloom, Black, and Lipsey (2008) argued that effect sizes should be interpreted with respect to empirical benchmarks that are relevant to the intervention, target population, and outcome measure being considered. In particular, these benchmarks can include: (a) normative expectations for change, (b) policy-relevant performance gaps, and (c) effect size results from similar studies. Hill and colleagues (2008) illustrated these benchmarks regarding academic achievement. However, there are very few comparable studies in social behavioral domains (Dong et al., 2016).
Purpose
The purpose of this study is to provide reference values of these design parameters by analyzing data from eight large scaled intervention studies. In particular, we are going to estimate: (1) the empirical benchmark of meaningful effect sizes regarding the policy-relevant demographic performance gaps (race/ethnicity, gender, and SES); (2) ICC for schools; and (3) R-squared at different levels (school and student). We will focus on 2-level multilevel randomized trials with students (level 1) nested within schools (level 2).
Method
Data. Data for this study came from eight IES funded multilevel randomized trials evaluating the effectiveness of school-based prevention interventions. The analytic sample includes 35,684 students from kindergarten to Grade 8 in 239 schools in 4 states (Maryland, Missouri, Virginia, and Texas). The student sample is relatively diverse: White (40.8%), Black (48.9%), Hispanic (5.7%); female (48.1%); eligible for free or reduced price lunch (51.5%).
Variables. The Teacher Observation of Classroom Adaptation-Checklist (TOCA-C; Koth, Bradshaw, & Leaf, 2009) is a nonclinical measure of children’s behavior completed by teachers. Subscales of the TOCA-C include concentration problems, disruptive behavior, emotion dysregulation, family involvement, family problems, internalization, and prosocial behaviors. The psychometric properties of the TOCA have been well documented (Bradshaw & Kush, 2019; Koth et al., 2009). The other relevant variables used in the analyses include student grade, race/ethnicity, gender, status of free or reduced-price lunch, and treatment status.
Analytic Plan. We calculated design parameters and the benchmarks regarding the policy-relevant demographic performance gaps/disparities before and after receiving the interventions using 2-level models with students nested within schools.
Results:
Some preliminary results reported in Table 1 and Figures 1 & 2.

Authors

• Nianbo Dong, University of North Carolina - Chapel Hill

• Keith Herman, University of Missouri

• Wendy Reinke, University of Missouri

• Sandra Jo Wilson, Abt Associates Inc.