Using and Interpreting Effect Sizes

• In Event: Disrupting Hegemonic Quantitative Practice

Abstract

The vast majority of studies on college student impact emphasize the statistical significance of their findings, while paying little or no attention to the magnitude of the results. Unfortunately, this overreliance on statistical results means that some findings with virtually no practical significance are treated as being “important.” As one illustration of this problem, some large secondary datasets regularly contain at least 10,000 students, which then leads to considerable statistical power. With that sample size, a trivial correlation of r = .02 would be significant at the traditional cutoff of p < .05. Few people would believe that such a small association should lead to meaningful implications for practice, especially if a substantial financial investment would be required to enact change. This overemphasis on p-values can also lead to erroneous conclusions for subgroup analyses, since the same effect size is more likely to be statistically significant for numerical majority groups (for whom more statistical power is available) than for numerical minority groups.

If the purpose of college impact research is to improve outcomes for all students and society, then understanding the magnitude of impact is crucial for determining the effective investment of resources by colleges and universities, state and federal governments, and private foundations. However, no shared understanding of what constitutes a “small,” “medium,” or “large” effect currently exists in higher education. Cohen (1988) has provided general guidelines for effect size metrics for all of social science research, but he repeatedly cautions that these should be (re)considered within their appropriate context. According to leading scholars and organizations (see Valentine & Cooper, 2003; What Works Clearinghouse, 2014), the recommended values should be smaller for educational research, because measurement error reduces the observed relationship between variables, and interventions or practices may not be implemented as intended.

This presentation will provide a discussion of these issues as well as offer guidelines for three effect size metrics that are applicable to many higher education studies: (1) Cohen’s d (or standardized mean difference), (2) standardized regression coefficient, and (3) average marginal effect (the same recommendations also apply to delta-p). Considerations for the appropriate use of these guidelines will be provided.

Author

• Nicholas A. Bowman, University of Iowa