Identifying When Mindset Interventions Are Effective for Students: A Two-Model Machine Learning Approach

• In Event: How Context and Learning Mindsets Advance Academic Outcomes: Findings from National Study of Learning Mindsets

Abstract

Objectives:
Students adopt learning mindsets, which are reflected in their approaches to learning new and difficult material (Kizilcec & Halawa, 2015). The national study of learning mindsets (NSLM) was a large-scale intervention study on growth mindsets (where students believe they can improve their intelligence through effort) and fixed mindsets (where students believe their intelligence cannot improve). In this paper, we employ machine learning methods to uncover student characteristics that predict whether (and how well) the NSLM intervention works for different students.
Perspectives or theoretical framework:
Recent research shows that growth mindsets can lessen the negative relationship between poverty and academic outcomes (Claro, Paunesku, & Dweck, 2016), and have positive effects on learning (O’Rourke, Haimovitz, Ballweber, Dweck, & Popović, 2014; Yeager et al., 2016). However, little is known about specific student characteristics (e.g., racial identity, personality traits) that predict when growth mindset interventions will work. Thus far, efforts to discover such characteristics have focused on predominately linear, theoretically-motivated relationships (e.g., Claro et al., 2016; Yeager et al., 2016). We approach this problem from a data-driven perspective, enabled by a large dataset and cross-validation to avoid false discovery errors.
Methods:
We approached the problem of discovering which student characteristics predict mindset intervention efficacy by building two machine learning models, each trained with 73 student-level variables including demographics and psychological measures. These models were tree-based regression models (Chen & Guestrin, 2016), allowing discovery of non-linear relationships and complex interactions between variables. We trained model 1 to predict difference in GPA before and after the intervention (i.e., GPA improvement) in the control condition. This model was then applied to students in the treatment condition, thereby estimating how much their GPA would have improved – based on student characteristics – had they not been given the growth mindset intervention. Finally, we trained model 2 to predict the difference between estimated and actual GPA.
Data:
We examine the NSLM dataset at the student level. The NSLM was a randomized controlled trial in which students in the treatment condition were given an intervention designed to induce a growth mindset, with grade point average (GPA), demographics, and other measures collected over two timepoints for both treatment and control conditions. This dataset includes 22,690 students from a nationally-representative sample of 76 public U.S. high schools. improvement – this difference being attributable to the intervention and unobserved variables.
Results:
Our results show medium prediction accuracy for model 1 (r = .444), indicating that student characteristics were related to GPA improvement. Important predictors in model 1 include demographic variables like gender and racial identities, psychological measures like mindsets, and prior grades. The full paper will focus on student characteristics that predict intervention efficacy (model 2).
Significance:
Machine learning has potential to uncover complex, non-linear predictors of intervention efficacy, thereby enabling more precise, targeted interventions for students who will benefit from them. Moreover, results highlight students and groups of students for whom a mindset intervention is unlikely to work well; in these cases, future work is needed to develop new strategies that are more effective.

Author

• Nigel Bosch, University of Illinois at Urbana-Champaign