Algorithmic Tradeoffs, Applied NLP, and the State-of-the-Art Fallacy

• In Event: 30802 - Machine Learning and Classical Statistical Approaches: Trade-offs, Integration, and Debate (Co-sponsored by Section on Mathematical Sociology)

Abstract

Computational sociology is a growing area of research, but the computational tools used for analysis vary greatly in their power, transparency, and usage. It is also the case that methods developed in computational science become popularized once they are able to surpass shared benchmarks of classification and prediction accuracy. In this paper, we consider how these dynamics might influence computational method selection in sociology and subsequently influence sociologists to select methods that do not best answer or fit their research question. To demonstrate our point, we compare results from a previous natural language processing (NLP) driven analysis of college admissions essays (n = 240,000) using more recent and older methods. The methods are also distinct in their relative statistical flexibility and opacity. We find that the more newer, more sophisticated methods which surpassed the older, more transparent methods in past computational benchmarks and popularity did not exceed the performance reported in the original study. These results suggest the latest computational tools might not always be appropriate by default. We also consider the social meaning and implications of our findings to future research on text analysis and authorship characteristics. Our findings inform similar studies of text but also sociological adoption of NLP methods.

Authors

• AJ Alvero, Cornell University

• Ruohong Dong, University of Arizona

• Klint Kanopka, New York University

• David Lang, Stanford University