Questioning the Dogma of Statistical Significance to Move Beyond p-Values in Equity Research

• In Event: AERA Roundtable Session Thursday 9:50 am Four Seasons Ballroom 1
  In Roundtable Session: Challenging Traditional Norms in Quantitative Research: Critical Perspectives on Equity, Racism, and Statistical Pedagogy (Table 5)

Abstract

Few ideas are as dogmatic in education research as p < 0.05. Despite the statistics community’s calls to move beyond this threshold (Wasserstein, Schirm, and Lazar, Am. Stat., 2019), problematic uses of p-values persist. Researchers often misuse p-values as go-no-go tests and misinterpret them as indicators of educational significance. We reviewed ten articles on equity in physics instruction, examining their use of p-values, effect sizes, and uncertainty. Our results indicated consistent misuse of p-values. We include three examples using simulated data to illustrate how studies dismissed variables equivalent to those they presented as important. The analyses sometimes either added variables to models or dissagregated data until complicating demographic variables became non-significant.

Authors

• Jayson Nissen, Montana State University

• Ben Van Dusen, Iowa State University