Crosswalking Data Collection: Mis/alignment of Quantitative Variable Selection and Qualitative Interviews Questions

• In Event: Design-Strengthening Strategies for Longitudinal and Multilevel Mixed-Methods Studies of Education Systems

Abstract

The objective of this presentation is to demonstrate the advantage of using a complementarity strategy for developing intentionally mis/aligned crosswalks between quantitative and qualitative data collection tools.
In the MM literature, complementarity is recognized as a plausible rationale for using mixed methods, sequentially or concurrently. Complementarity is the rationale for using MM when the different approaches are intended to address overlapping and/or distinct aspects in a way that generates deeper and/or broader understandings.
The planned analyses to address RQ3 (How do schools influence students’ course trajectories, and what data/processes do schools use to promote college readiness?) included hierarchical linear models (HLM) and case studies. We planned HLM analyses to produce indicators, for example, of whether students from any of in-state high schools tend to exceed or fall short of post-secondary expectations despite similarities in terms of high school academic achievement variables. We planned case studies (Plano Clark et al., 2018) of a subset of high schools with the aim of generating rich descriptions of value-added college-readiness strategies.
Given the COVID-era challenges in schools across our state, limiting participation burden on school staff was important. We employed a complementarity strategy to reduce the High School Interviews (see Table 1) but maintain is usefulness as a case selection support tool.
The interviews were initially conceptualized as audio/video conferences, taking 30-60 minutes and soliciting information about academic courses, instructional materials, supplemental programs, and enrollment/eligibility policies. We identified alternative ways to collect relevant information in advance of the interviews. For example, review of the qualitative course catalog data and quantitative course enrollment data revealed a unique five-year dual enrollment program some schools offered in partnership with regional institute of higher education. Keeping other data sources in mind, we iteratively revised the statewide school interview protocol using these guidelines for data source mis/alignment:
1. Focus on questions that cannot be addressed in any way by other data sources.
2. Develop questions aimed at extending understanding of emergent school profiles.
3. Limit confirmatory triangulation questions.
For example, instead of asking about levels (e.g., college-preparatory vs. honors) of math courses offered, we asked: What are some examples of online learning programs your school uses to support math instruction? Also, instead of asking what course pathways the school offers, we asked: What is your school’s process for recommending courses to students?
This example illustrates the economical nature of complementarity strategies. There are clear logistical advances. It can reduce person-hour requirements for implementation while enhancing the interpretability of overall results of a MM study. In our case, the revised protocol allowed us to compare available and traveled course trajectories in the contexts of schools’ logic for making various recommendations to students. As a result, differences in schools’ ideas about how college-readiness can/should be supported among their student population may be considered in case selection. This improves the chances of selecting cases that illustrate an array of value-added college-readiness strategies applicable in a wider range of schools.

Authors

• Katrina K. Morrison, University of Delaware
  Presenting Author

• Marcia Gail Headley, University of Delaware
  Non-Presenting Author