Measured Success: Evaluators as Insiders, Outsiders, and Knowledge Producers

• In Event: STS and Universities - II

Abstract

ABSTRACT
Evaluation is a rapidly growing field. The number of professional evaluation associations exploded from two in 1981 to currently 173 organizations with 53,000 members world- wide. While STS has noted the turn toward quantification and reporting in science, including in universities, there has been little research on how the people doing the work--evaluators themselves--navigate the call to deliver objective measures of complex social processes. This paper investigates the role evaluators see themselves playing in knowledge production, and the puzzles their work raises for scholarship on scientific work in universities. The data come from 30 interviews with paid evaluators of US projects funded since 2015 by the NSF’s NRT program (for interdisciplinary graduate training, successor to the IGERT program). The NRT solicitation requires evaluation of every training grant. Evaluators occupy a variety of employment positions across the projects including: independent contractors, employees in evaluation firms, and colleagues at the same university as the NRT PI. Evaluators must balance insider roles advising PIs with outsider roles in reporting to NSF on the effectiveness of the funded project activities. As befits a professionalizing field, evaluation is far from standardized and rife with tensions. In navigating the inherent tensions, evaluators are reshaping what the work and profession of evaluation actually is, at times at odds with discourses of knowledge production surrounding evaluation. As evaluators become ever more present experts, the need for STS research that better reflects both the daily work and professionalization of evaluation is clear.

Authors

• Elisa Martinez, University of Massachusetts

• Laurel Smith-Doerr, University of Massachusetts

• Tim Sacco, University Of Massachusetts Amherst