Critical Possibilities: Interdisciplinary Recommendations on Using Secondary Data in Education for Social Justice

• In Event: Doing Critical Quantitative Research: Methodological Commitments to Equity and Justice

Abstract

Secondary data in education, defined here as quantitative data not created by the researcher, often exists in the form of administrative or larger datasets from organizations or governmental agencies (e.g., NCES). These datasets can provide opportunities to generalize at the population level, examine heterogeneity between or within groups, and inform policy- and practice-relevant decisions (Figlio et al., 2015; Bryan et al., 2017). However, while these larger datasets can help us understand demographic trends, these data can help maintain or support racist, sexist, classist, ableist, and other problematic ideas as well. For example, data can be “dirty” in that the data collected reinforce biases or are simply inaccurate (e.g., Richardson et al., 2019) or be used to reinforce bigotry (Covarrubias & Vélez, 2013). From the beginning, major statisticians in the field (e.g., Galton, Pearson, and Fisher) promoted eugenics and used statistical methods to try and justify their ideas (Cleather, 2020). Therefore, researchers must be vigilant in using secondary data sources in identifying and reducing white supremacy in research.

Purpose

This paper will synthesize recommendations from across the educational field, such as those for administrative educational data (e.g., Viano & Baker, 2020), along with broader, theoretical recommendations on “How to QuantCrit” (Castillo & Gillborn, 2022) as they apply to secondary data. The paper will be structured through the phases of a research study although many concepts could be used throughout the entire research process. Each of the following points will be elaborated on in the final paper and presentation citing scholars in the field and examples but only listed here due to space constraints.

Before Analysis

Before starting on a secondary data analysis project, researchers need to choose theoretical frameworks that acknowledge systems of oppression; form justice-oriented research questions; select variables that speak to structures and add nuance; state their positionality as it relates to the study and the dataset; request community feedback on approach; and cite/base all work on research that reflects the population of focus.

During Analysis

After all of the preceding work, researchers need to create and justify reference groups and sample sizes; and examine heterogeneity to go beyond means. Analyses should dive deeply into the data to uncover the full story of participants along with the limitations of the dataset. Researchers must balance theoretical and methodological integrity rather than rely on outdated norms (e.g., the superiority of inferential statistics over descriptive statistics with small samples).

After Analysis

Once researchers have the results from their final models, they should follow traditional best practices for quantitative methods such as reporting effect sizes, stating limitations, and visualizing data but should frame these practices using QuantCrit, paying attention to not further recreate harms. Researchers must add as much context as possible to create a narrative that can then create change via policymakers, practitioners, and dataset creators. To do this, researchers must rethink traditional publishing methods to ensure results make a difference and include the populations of focus.

Author

• Rachel L. Renbarger, FHI 360