Peer Review as Goal Alignment: The Case of Computer Science Research

• In Event: Mathematical Sociology Section Flash Talks
  In Refereed Roundtable (60 minute): Table 7. Peer Review as Goal Alignment: The Case of Computer Science Research

Abstract

Academic peer review is an important site in the social production of scientific knowledge. Social factors, such as author prestige and schools of thought, influence peer reviewers and peer review outcomes. I identify and explore patterns in the actual content of peer reviewer comments as a step toward understanding how that feedback affects publication. Using Discourse Atom Topic Modeling, a topic-modeling method based on word embeddings, I extract topics from thousands of reviewer reports in a major computer science conference. I find both praise and criticism for various aspects of manuscripts, from writing style to research design, and connect these topics to quantitative reviewer ratings. From this analysis there is evidence that peer review functions to reinforce the norms, goals, and benchmarks of the field. I encourage further research to understand how these processes differ across academic fields, as well as in other steps in the publication process, such as editorial decision-making.

Author

• Christina Wilmot, University of California-Los Angeles