When You Come to a Fork in the Road, Take It: Rashomon Sets for Sociologists

• In Event: Mathematical Sociology Section Flash Talks
  In Refereed Roundtable (60 minute): Table 13. When You Come to a Fork in the Road, Take It: Rashomon Sets for Sociologists

Abstract

Social science researchers have developed many statistical methods for determining which models to use for social inquiry. Conventional approaches rely on model selection, averaging over many models, or "multiverse" techniques that assess robustness across many researcher-defined alternative specifications. We argue that these methods overlook the fundamental ambiguity inherent in data: even when researcher decisions are perfectly standardized, many distinct models can fit the data equally well. We propose addressing this "model multiplicity" through the lens of the "Rashomon effect," a term coined by Breiman (2001) to describe situations where many statistical models that imply different mechanistic explanations have near-equivalent performance over the same data. Recent advances in computational methods have created tools to completely enumerate full sets of such models for important model classes, called "Rashomon sets." We demonstrate the inevitability of model multiplicity with a simulation study. We offer a set of post-fit criteria researchers can use to navigate model multiplicity through Rashomon sets, then demonstrate their utility with an empirical application, showing how different aggregations of religious categories yield distinct plausible explanations for life satisfaction. We contend that Rashomon set methods offer uniquely exhaustive opportunities for social inquiry, promoting more intellectually honest model selection while maintaining rigor.

Authors

• Adam Visokay, University of Washington

• Edward Lucas Hock, University of Washington

• Kentaro Hoffman, University of Washington

• Tyler McCormick, University of Washington