Bibliometric network analysis: Methodological challenges and approaches for the study of policy knowledge

• In Event: Policy knowledge networks and lesson drawing in an era of international comparison

Proposal

The purpose of the Nordic study is to identify and analyze the knowledge and evidence that is used by educational policy makers in five countries: Denmark, Finland, Iceland, Norway, and Sweden. Paraphrasing Pascual-Nebreda et al. (2018), I argue that knowing the knowledge base of recent education reforms is essential to understand the “nutrients” used for the development of the policy. To do so, our research team is applying bibliometric network analysis techniques to official policy documents. Specifically, we examine in each country a set of white papers published by the government (i.e., source documents) and the citations to other resources (references). In addition to examining the composition of the references (e.g., year of publication, location of publisher [local, regional, foreign], and type of publication [peer reviewed journal, report by international organization]), we also explore the nature of the relationships between source documents and references. For example, we investigate the popularity of certain references (i.e., how many times each reference was cited). We also examine the extent to which different source documents draw on similar references, and the patterns of co-citation of references (i.e., the frequency with which two references are cited together).

The methodology for this project draws on past research in the sociology of science and knowledge. Holman, Lynch, and Reeves (2017) used bibliometric network analysis of 200 published articles to explore which disciplines and concepts dominate the field of Health Behaviour Interventions. Pascual-Nebreda et al. (2018) used articles published in the American Behavioral Scientist (n=3,689) to examine the intellectual structure of the journal. Using organizational membership data, Daipha (2001) and Pizmony-Levy (2016) examine the social structure of academic/professional associations (through co-membership patterns).

In this presentation I will discuss four challenges in the application of social network analysis to the case of policy documents. The first challenge is data availability. Scholars conducting bibliometric network analysis of scientific publications often obtain data from the Social Science Citation Index (SSCI). For our project, however, we need to select the source documents and manually generate the references dataset. This requires setting up detailed data cleaning procedures. The second challenge is sampling strategy as these decisions have serious implications for the analysis. For example, using a sample of official policy documents rather than a sample of policy documents produced by different stakeholders (Brøgger et al., 2016) limits us to certain policy knowledge. The third challenge is the unit of analysis. Although the data is collected at the document level (source document citing reference documents), it is possible to analyze the data at the author/publisher level to examine patterns of “echo chambering”. The fourth challenge, which is common in every comparative study, is how to standardize the method across countries and time.

The primary aim of this presentation is to consider the methodological challenges that our team is facing and to demonstrate our approach to address some of these challenges. The same methodological approach could be applied to other countries and debates about evidence-based actions.

Author

• Oren Pizmony-Levy, Teachers College, Columbia University