The Sensitivity of Small Area Estimates for Generalization

• In Event: Excellence in Education Research: Early Career Scholars and Their Work

Abstract

The Sensitivity of Small Area Estimates for Generalization

Over the past decade, statisticians have developed methods to improve generalizations from nonrandom samples using propensity score methods. While these methods contribute to generalization research, their effectiveness is limited by small sample sizes. Small area estimation is a class of model-based methods that address the imprecision due to small samples and prior research has shown its potential to improve precision in generalization studies with small samples. However, small area estimates are sensitive to model assumptions and within the generalization framework, they are also sensitive to the assumptions of the propensity score model. In this study, we assess the extent to which violations in the assumptions of propensity score methods impact the effectiveness of small area estimates to improve generalizations. Specifically, we examine how violations in the sampling ignorability assumption for propensity scores affect the performance of small area estimators. Using a simulation study, we vary the extent to which sampling ignorability is violated and compare the performance of four small area estimators in relation to the traditional design-based estimator used in generalization studies. We identify the conditions, if any, under which the small area estimators outperform the design-based estimator when sampling ignorability is violated. We provide an application of small area methods using an empirical example based on the Study of Deeper Learning and discuss the implications of violations in assumptions on estimators in generalization studies.

Author

• Wendy Chan, The University of Pennsylvania