AI and the nature human judgment in science

• In Event: Science, Knowledge & Technology Section Roundtables
  In Refereed Roundtable (60 minute): Table 5: Knowledge Production in the Age of AI

Abstract

Innovation in science and technology depends on our values. What we judge to be true, new, excellent, elegant, or useful helps determine what gets published, funded, awarded, and patented. But observing our values at scale has been difficult. And tracing the ways they actually shape new ideas and inventions harder still. In this paper, we draw from theories in the philosophy of science and leverage a variety of Artificial Intelligence (AI) tools to argue and demonstrate a new way of viewing human expertise in evaluation. Based on our AI-augmented study of the peer review proceedings of two scientific venues, our preliminary findings suggest plurality in values (versus one-dimensionality), emergence and liminality in science and inventions (versus completeness and definitiveness), and intersubjectivity and collaboration in evaluation (versus unilateral judgment or algorithmic decision making). Together, we find evaluation is an inescapably social interchange, where scientists and inventors interact with their evaluators to iteratively refine their ideas and technologies so that they land — in impactful journals and as valuable products. Thus, by way of our principled and supervised adoption of AI techniques, we center human experts and human values in evaluation and theoretically scrutinize calls for full substitution and automation of the production and evaluation of science.

Authors

• Daniel Scott Smith, Duke University

• Daniel A. McFarland, Stanford University