Empirical Benchmarks for Interpreting Effect Sizes of Social and Behavioral Outcomes

• In Event: Improving the Design of Program Evaluations in Education and Prevention Science

Abstract

Social and behavioral measures such as aggressiveness or social competence are common in education research as outcomes of interest. Such outcomes are important indicators of student adjustment and impact estimates on the effectiveness of interventions are increasingly reported in the form of effect sizes.

Effect sizes can help researchers make comparisons across studies, but they are not always accessible. It can be challenging to determine what is a meaningful difference on a social or behavioral outcome. The traditional small-medium-large distinctions commonly used to ease interpretability are not very useful for decision-makers (a “small” effect size on a critical outcome may be more meaningful than a “medium” effect size on something else). Easily understandable options for interpreting effect sizes on social and behavioral outcomes can help researchers effectively communicate the findings of their research.

Purpose
The purpose of this paper is to provide researchers and consumers of research with several options for effect size interpretation for social and behavioral outcomes. We address three questions:

1. What are normative changes in social and behavioral outcomes over time? Do they vary by age or grade level, type of social or behavioral outcome, or informant?
2. How does the effect of interventions on social and behavioral outcomes differ by demographic subgroup (race/ethnicity, gender, age, and socio-economic status)?
3. What is the average effect size for social and behavioral outcomes for certain types of interventions and populations?

Methods and Data Sources
This paper is based on a large-scale meta-analysis of school-based intervention studies reporting effects on social and behavioral outcomes. Eligible studies used randomized controlled trial designs, must have included an intervention delivered in a school setting, and must have targeted U.S. students in grades pre-K through 12.

The meta-analysis includes outcomes in several social and behavioral outcome domains: antisocial behavior, behavioral/emotional functioning, and social functioning. Means and relevant descriptive statistics were recorded from each study. This permits calculation of different effect sizes indices for examining changes over time in treated and untreated groups, differences in outcomes for different subgroups, and differences between conditions on the outcomes over time.

Results
To address Question 1, we will report mean change effect sizes for a range of measurement intervals for the comparison group conditions. These will serve as benchmarks of normative gains on particular outcome domains over various intervals which can be used by researchers to compare against intervention impacts.

To address Question 2, we will report pretest and posttest differences between subgroups, as well as differences in the posttest gap between intervention and comparison conditions by subgroup. These estimates can serve as benchmarks for researchers who may be interested in comparing whether an intervention reduces “behavioral gaps” between different groups.

To address Question 3, we will calculate average effect sizes for studies grouped by program, prevention strategy, format, grade level, and dosage/duration. These estimates can be used in power analyses and may be useful for researchers who want to compare intervention impacts against the range of effect sizes observed across other studies.

Authors

• Sandra Jo Wilson, Abt Associates Inc.

• Brian Freeman