Ensemble 3: Dissecting Views of Learning in Risk Algorithms

• In Event: Expansive Futures for Disability Intersectional Learning Research Braiding Culture, History, Equity, and Enabling Technologies

Abstract

Predictive models to prevent school failure: Robert Balfanz (Johns Hopkins University);

Algorithmic justice: Ruha Benjamin (Princeton University)

Digital tools are increasingly used to categorize students along a “risk” taxonomy for guiding resource allocation and preventing dropout and school failure. The algorithms used to build these infrastructures in school districts draw from longitudinal datasets and increasingly guide decision making in placing and treating groups of students with “high-risk” profiles. For instance, schools or districts may look for early warning indicators typically indexed in the accumulation of signs such as low grades or reduced test scores in a core subject, suspensions, and absenteeism. In some locations, these algorithms take into account contextual issues such as student behavioral history to adjust interventions. In other places, decision-making systems include a police officer or a school resource officer (SRO) in the decision-making team; thus, forestalling the consequential impact these systems can have on students’ trajectories. The underlying algorithms of these information infrastructures are regarded as objective, neutral and accurate.
5
Nevertheless, we note these indicator systems rest on assumptions about student learning, competence and engagement that are rarely inspected. But recent scholarship in social studies of science and sociology is raising profound equity questions. To illustrate, how can the algorithms used to build early warning predictions potentially exacerbate or hide inequities under a semblance of neutrality and altruism? Benjamin (2019a) described these concerns with the term “the New Jim Code”—"automated systems that hide, speed, and deepen racial discrimination behind a veneer of technical neutrality” (Benjamin, 2019a, p. 422). Instances are surfacing in medicine and other industries/sectors (e.g., housing, policing, criminal justice). For example, Obermeyer et al. (2019) examined a health-risk predictive algorithm designed to determine patients’ health needs using “cost of care” as a proxy. Obermeyer et al. found that African American patients labeled with the same risk level as White individuals had greater health needs, mainly because systems spend less on their health care. Benjamin (2019a) noted: “This study contributes greatly to a more socially conscious approach to technology development, demonstrating how a seemingly benign choice of label (that is, health cost) initiates a process with potentially life-threatening results” (p. 421). Further, this study documents the importance of counting not only the number of health conditions, but also measuring their severity. Of significance, the study calls attention to the potential consequences of building predictive models using algorithms that neither theorize nor account for systemic racism.
Following this reasoning, we acknowledge that educational policies and practices are enveloped in systems of racial stratification. Thus, the concerns about algorithmic justice emerging in other areas apply to the education field. It is necessary that we demystify these technologies and work to enhance the transparency of the algorithms used to build systems that despite good intentions might perpetuate inequalities. As these tracking systems continue to flag groups that have been historically underserved because of their race, social class, language, national origin, or gender, it is crucial we unveil the mechanisms that produce disparate outcomes and their underlying visions of competence and learning. The data used to create automated systems is characteristically historical. We know such history in education entails racial segregation, unequal funding patterns, racial and socioeconomic gaps in opportunities to access well prepared teachers and rigorous curricula, as well as limited access to an array of support services. If algorithms do not account for these sociohistorical dimensions of students’ pathways, the designs of automated systems could end up encoding inequalities and sanctioning injustices (Benjamin, 2019b).
We will invite two scholars to focus on one automated system that produces warning signals about students’ probabilities to fail or drop out of school. The scholars will engage with the justice concerns outlined above. Inspired by the Algorithmic Justice initiative, we will ask these participants to grapple with questions such as [we will refine Qs in consultation with panelists]:
• What are the assumptions about learning in the system’s algorithms?
• What safeguards are available to audit the system for accuracy and fairness? How does the algorithm work with disparate populations?
• How does the algorithm account for racial inequalities and other injustices built in education systems?
• How can the system’s scores be counterbalanced with learners’ learning pathways across contexts and time?

Author

• Safiya Noble, University of California, Los Angeles