Paper Summary
Share...

Direct link:

Using Bibliometric Data for Program Evaluation: A Citation Analysis of the National Science Foundation’s Program on Research and Evaluation on Education in Science and Engineering

Fri, April 13, 2:15 to 3:45pm, Marriott Pinnacle, Floor: Third Level, Pinnacle III

Abstract

Counting and weighting citations to journal articles is a routine, if sometimes controversial, method for evaluating research output. Though typically used to evaluate individual researchers, citation analysis has also been applied to research centers and university departments. Rarely, however, has bibliometric data been used to evaluate entire programs of research. In collaboration with NSF, the Center for Advancing Research and Communication in STEM (ARC) is conducting a citation analysis of publications produced by REESE projects in order to understand the impact of this large, federally funded education research program on knowledge building within the academy.

Data Sources and Methods

REESE funded, in whole or in part, a total of 346 projects from September 2006 through December 2010. Project publications likely to have been supported by this funding were identified by asking REESE PIs on these projects to send lists of journal articles, books, book chapters, and conference proceedings that resulted from their NSF funding. Overall, 86% of REESE projects responded to this request or had such publications available through FastLane. Additional publications were then obtained by searching for all 346 NSF award numbers in Google Scholar, yielding a “virtual” response rate of 100%. Both Google Scholar and Web of Science were used to obtain citation counts for each publication, and the Eigenfactor™ Article Influence score for each journal also was recorded where available. Finally, these bibliometric data were merged with project demographic information available through FastLane and from structured abstract forms that REESE projects routinely submit to ARC. Both bivariate and multivariate statistically analyses were performed using the number of citations per publication as the primary dependent variable.

Results and Significance

A total of 872 published research products were identified from 346 REESE projects, with an average of nearly five citations per publication in Google Scholar by May 2011. Statistical analyses suggest that REESE synthesis studies and projects employing insights from the behavioral and cognitive sciences have significantly higher mean citation rates. Conversely, citations per publication are significantly lower for projects that use descriptive methods or include a focus on mathematics. Using Web of Science results only, the average article influence (AI) score of REESE journal publications is 2.96, well above the normalized average of 1.0 across all journals in the Web of Science. Since journals in education, psychology, and sociology have below average influence overall, this suggests that REESE projects are having a significantly higher-than-average impact on educational research in the U.S. This analysis also shows that it may be possible to increase the academic impact of REESE projects by allocating program funds among projects that employ certain research methodologies or focus on specific topics. Based on these results, we conclude that citation analysis is a valuable tool for both evaluating and managing programs of funded research in education and beyond.

Authors