The "Students-at-Risk" Algorithm in a Higher Education Platform

• In Event: Platform Pedagogies: Examining the Public Good in Learning, Schooling, and Educating Across Emerging Platforms

Abstract

Objectives and theoretical frameworks
In this paper I examine an attempt to integrate machine learning algorithms in an educational platform: an open source Learning Management System (LMS). While they certainly involve computation, the actual meanings of algorithms emerge dynamically through relations between actors, institutions and, increasingly, platforms (Bucher, 2018; Kitchin, 2014). Drawing on this theoretical position, the article describes a specific LMS as an “algorithmic assemblage” and considers a broad algorithmic process, enacted through the LMS, as its unit of analysis:

if a student’s historical pattern of online activity is classified as inadequate, then the student will be predicted to be “at risk”, and a remediation action will be instantiated.

Methods and data sources
The students-at-risk algorithm was analysed as it developed and was negotiated at the interface between the LMS and a large Australian university. These two settings are identified with pseudonyms: Open Source Learning (OSL) and Southern Victoria University (SVU). In-depth interviews were carried out with a small number of strategic “super informants” (Levi Strauss, 1963) working in technical and educational roles. These include:

● the data scientist who laid the groundwork for OSL’ s machine learning integration;
● the main learning analytics researcher at OSL;
● OSL’ s Business Development Manager who coordinates relationships with senior administrative staff in universities;
● four individuals working in leadership and administrative capacities in the “Education Innovation Centre” at SVU.
In addition, multiple “digital traces” were collected. These traces are discoverable on the internet thanks to the open-source nature of OSL. They comprise sections of digital code, data architecture diagrams and forum discussions. This combined human/code methodology allowed the study to attend, simultaneously, to the technical and the social dimensions.

Results and significance
The students-at-risk algorithm was at the centre of a process of sociotechnical construction: a rather messy array of computational design choices and human interactions where actors – no matter the level of technical expertise - only had a partial understanding of the underlying logics. As a result, the meaning of predictive modelling in relation to key constructs in the algorithm (“risk”, “remedial action” and “online activity”) was constantly reconfigured in an inductive and iterative fashion, i.e. proceeding from unwarranted, heuristic assumptions treated as self-evident:

- extensive data collection is inevitable;
- large datasets contain valuable, yet “hidden”, information about learning;
- patterns of online activity are indicators of future educational failure.

Each of these assumptions is, in fact, probabilistic and uncertain in nature. This uncertainty shaped a discourse where every aspect of the algorithm was contested.

This study points to a broader problem: the pursuit of predictive knowledge in higher education is manufacturing “heuristic decision spaces”, effectively represented by the students-at-risk algorithm. A heuristic decision space is one where actors operate with a limited comprehension of the underlying logics, and therefore proceed inductively from what they assume to be self-evident “truths” (Miller, 1994). As a result, pedagogic understandings are developed in a very instrumentalist and ad-hoc fashion, according to pre-existing frames of reference and biases that shape the nature of “platform pedagogies” in unreflective ways.

Author

• Carlo Perrotta, Monash University