An Empirical Evaluation of Methods for Detection of Effect Modification

• In Event: Excellence in Education Research: Early Career Scholars and Their Work

Abstract

An Empirical Evaluation of Methods for Detection of Effect Modification

Widespread access to high-quality educational data sets has made it easier for educational researchers to search for answers to causal research questions with non-randomized data through conditioning strategies. Recent advances in statistics and causal inference have focused on understanding conditional average treatment effects, but methods for detecting and describing treatment effect heterogeneity with observational data have not entered the mainstream in education or behavioral sciences. We explore, through Monte Carlo simulation using data generation parameters motivated by ECLS data, the ability of emerging approaches highlighted in recent work to (a) accurately estimate individual treatment effects, (b) detect treatment effect heterogeneity when it exists (power), and (c) correctly conclude there is no heterogeneity when it does not exist (Type I error rate).

Author

• Bryan Keller, Teachers College, Columbia University