Verifying Qualitative Research

• In Event: Challenges in Qualitative Data Collection and Analysis

Abstract

Qualitative research – generating and analyzing qualitative data – makes significant contributions to social inquiry. Extraordinary insights can be gained from qualitative data produced through interviews or archival research, for example. Analytic methods such as process tracing and Qualitative Comparative Analysis (QCA) can be used to generate causal inferences independently, or in combination with other methods of analysis. In sequential multi-method research, qualitative work can elucidate the causal mechanisms underlying a previously established causal relationship. In integrative multi-method inquiry, qualitative work can play an even more central role in causal identification.

The contributions that any research makes to the generation of knowledge are contingent on whether the author followed the rules that guide the techniques for generating data, and the methods for analyzing data, that she employed. The possibility of “reappraising” (Gerring 2020) the truth-value of a researcher’s findings depends on (1) the rules for data generation and analysis being sufficiently well-specified that they can be followed; (2) the author providing the underlying data (to the extent permitted by law, ethics, and logistics) and describing how they were generated and analyzed to produce knowledge claims and draw conclusions; and (3) the existence of clear standards for assessing the author’s adherence to the relevant rules.

Replication and reproduction play a prominent role in the evaluation of quantitative research. To date, no similar standard has emerged for qualitative research. The heterogeneity of qualitative inquiry certainly complicates the development of stable, standard practices for reappraising such research: scholars who conduct qualitative inquiry have varying epistemological commitments, employ many different research designs, and use a broad variety of tools and techniques. Yet for the kinds of evidence-based qualitative scholarship where such standards might be helpful, such guidelines must be developed.

This paper offers an initial framework for developing appropriate standards for reappraising different types of evidence-based qualitative inquiry. In the most abstract sense, research involves three tasks: (1) collecting information and generating data; (2) analyzing those data; (3) drawing inferences and developing claims and drawing conclusions on the basis of that analysis. Reappraising any of those tasks might involve (A) assessing how the scholar carried out the task without redoing the task; (B) redoing the task using the same process, or (C) redoing the task using a different or revised process.

Reappraising a piece of scholarship thus entails making choices about which research tasks to assess and how to do so. For instance, for much research involving original qualitative data, re-generating the data (i.e., a combination of 1B) may be prohibitively expensive or time-consuming. Moreover, not all scholars who do qualitative work generate their own data, meaning reappraisal might begin with assessing how the author conducted the analysis. Further, the clarity and decisiveness of the rules guiding different techniques for generating qualitative data and methods for analyzing qualitative data vary. The less clear the rules, the greater the difficulty of discerning against what the conduct of the research should be compared.

How to reappraise a piece of scholarship also depends on how data are analyzed. With some methods (e.g., automated text analysis or QCA), data are aggregated in a matrix and analyzed holistically; reappraisal thus entails re-analyzing those aggregate data to see if the same overall inferences are drawn. With other methods, individual sources (e.g., interview transcripts, newspaper articles) are considered separately or in small groups, with cited sources serving as distinct inputs to the analysis. Here, reappraisal involves assessing the data-to-claim coincidence of dozens of micro-claims.

Drawing on Gerring’s (2020) framework for reappraisal, we focus initially on verification, i.e., re-analyzing (using the same method) the data an author originally collected/generated and seeing if the same results obtain. This is a promising entry point because verification represents the most straightforward type of reappraisal. If any form of reappraisal of qualitative research is feasible, verification should be; conversely, if it is unhelpful, more complex types of reappraisal are unlikely to make a contribution.

In the paper, we analyze ten articles employing diverse qualitative data and methods that were recently published in top political science journals, identifying or deriving their research designs, and considering how verification might proceed. Based on the results of that initial analysis, we offer an initial overarching framework that could be adopted by different research communities to help them to generate guidelines for verifying different types of qualitative inquiry.

Authors

• Colin Elman, Syracuse University

• Diana Kapiszewski, Georgetown University

• Robert M Demgenski, Syracuse University