Who Watches the Watchmen? Evaluation of Peer Reviews in Social Science Journals

• In Event: Advances in Measurement and Methodology

Abstract

The academic system of peer review has undergone many changes to its contemporary standard of using detailed comments and recommendations from at least two or more experts in a double-blind review process. This process, however, is not perfect due to systematic and idiosyncratic biases of journals, editors, and reviewers. In an attempt to increase the transparency of the review process, we launched an independent website that archives and aggregates feedback by users about their experiences with academic journals' review processes. We present in-progress analyses of N = 110 reports on descriptive statistics, as well as relationships between journal characteristics (e.g., impact factors) and their evaluations. This in-progress study’s contribution is threefold: 1) to evidence the utility of a public repository of authors’ review experiences as a check of journals’peer-review processes, 2) to identify trends and patterns in peer-review processes, and 3) to identify predictors of author perceptions of review quality.

Authors

• Malte Elson, University of Muenster

• James D. Ivory, Virginia Tech