Generalizing from Small Samples: Application to the Study of Deeper Learning

• In Event: Excellence in Education Research: Early Career Scholars and Their Work

Abstract

Policymakers and practitioners are often interested in knowing whether an intervention works and for whom it works. This interest in the generalizability of a study’s findings encompasses results from both randomized controlled trials, quasi-experimental studies, and observational studies. While statistical methods based on matching have made important strides in improving generalizations from randomized and non-randomized studies, less attention has been given to generalizations from small studies where the study sample comprises less than 5% of the inference population. While methods based on machine learning have been proposed to address the limitations of small sample sizes, these estimators do not necessarily converge when the sample to population size ratio is significantly small. In this study, I evaluate the extent to which machine learning methods, combined with optimal approaches to redefining broad inference populations, can be used to generalize from limited data. I explore several statistical approaches to redefining inference populations, based on propensity scores, and assess the extent to which estimates of average treatment effects, that are as bias-reduced and precise as possible, can be derived.

Author

• Wendy Chan, University of Pennsylvania