Advancing Rigor in Qualitative Case Study Analysis: Tensions and Trade-Offs

• In Event: Schools’ Experiences With School Improvement Grants (SIGs): First-Year Results From SIG Case Studies

Abstract

The main focus of the Study of School Turnaround is to document the change process in schools awarded a School Improvement Grant because of their chronically low performance. This methodological paper has two purposes: 1) to provide an overview of the data collection and analytical methods used in the study, and 2) to discuss advantages, tensions, and trade-offs in applying notions of rigor that derive from quantitative methods to a set of school case studies.
With respect to the first purpose, the paper will summarize the study’s data collection methods, which included in-person interviews with state, district, and school administrators; instructional coaches; teachers; parents; students (in high schools); and external providers; and surveys of teachers. At every stage of the data collection and analytic process, the study team applied procedures for quality control and inter-rater agreement. To ensure data quality, every interview transcript was reviewed by the data collection team before it could be coded; before engaging in the coding task, all researchers passed a test to ensure they would apply codes consistently; and senior researchers reviewed each coded document for accuracy. Throughout analyses, researchers documented standards of evidence and decision rules to create an “audit trail” and to enhance replicability. These are accepted standards for robust qualitative methods. In addition, however, the research team used specific behavioral indicators and more quantitative methods (including frequency thresholds and minimum number of respondents reporting each indicator) to define and code key constructs (such as levels of collaboration) used in the analyses. Every school “classification” was documented and independently reviewed by the site visit lead.
The advantages of this approach to quality control, documentation, and standards of evidence are that it enhances the rigor and face validity of the analyses, particularly for clients and reviewers used to more controlled and quantitative approaches. Indeed, qualitative researchers too often pay inadequate attention to such methodological concerns. Rather than explaining the assumptions and the standards of evidence underlying analyses, they often identify themes and generate findings in ways that could not be replicated by another researcher with similar access to the data. To avoid this problem, researchers in this study considered issues related to validity, reliability, and replication at every stage of data collection and analysis, often quantifying variables in uniform ways to enhance comparability across schools.
Such methods carry with them certain trade-offs, however. For example, although the researchers carefully identified, defined, and measured specific phenomena in each school, in doing so, we found it more challenging to retain a holistic focus on each school as a case, thus compromising one of the unique assets of case study methodology. This can be particularly problematic when the topic at hand is the change process in organizations as complex and context-specific as schools. This paper will examine such trade-offs and provide a methodological context for the papers that follow. We will then revisit issues associated with qualitative rigor in the discussion segment of the symposium.

Authors

• Jennifer A. O'Day, American Institutes for Research

• Beatrice F. Birman, American Institutes for Research