Measuring Mindset More Accurately With the Undergraduate Lay Theories of Abilities (ULTrA) Survey

• In Event: Broadening the Assessment of Intelligence Mindsets to Align With the Promises of Mindset Theory

Abstract

Theoretical Framework and Objectives
Students’ beliefs about whether their abilities can improve or are stable (i.e., mindset) act as a meaning system that influences how they perceive and respond to educational experiences (Blackwell et al., 2007; Robins & Pals, 2002). These effects have been leveraged to develop interventions that can dramatically improve student outcomes (Yeager et al., 2019). However, a recent meta-analysis revealed that effect sizes of mindset studies are highly variable (Sisk et al., 2018).

This inconsistency may be a consequence of measurement problems. The Implicit Theories of Intelligence Scale (ITIS) is the most widely used tool to measure mindset (Dweck, 1999). However, undergraduates interpret the items in the ITIS inconsistently (Limeri et al., 2019). Unreliable measurement could obscure the relationship between mindset and outcomes, causing replication issues. Following the measurement framework from Standards for Educational and Psychological Testing (AERA et al., 2014), we developed a new measure of mindset that is specific to the undergraduate STEM education context and thus more consistently interpreted by STEM undergraduates: the Undergraduate Lay Theories of Abilities (ULTrA) Survey.

Method
The development process was iterative and incorporated multiple sources of validity evidence. We interviewed 45 students to identify language that undergraduates would interpret consistently, collected feedback on the drafted items from 11 experts, and conducted think-aloud interviews with 29 undergraduates on the revised items. We administered revised items to 1,194 undergraduates at 64 institutions. Factor and item response theory analyses revealed that the internal structure aligned with expectations and all items had acceptable psychometric properties. In the current study, the final ULTrA survey was compared to the ITIS, in a survey given to 1,396 students from six institutions. To compare predictive effectiveness, we measured constructs related to mindset (belonging, evaluative concern, goal orientation, self-handicapping) and outcomes (intent to persist in science and grades collected from instructors after the semester ended).

Results
ULTrA had stronger predictive efficacy than ITIS; it correlated more strongly than ITIS with all of the constructs we measured. ITIS was not significantly correlated with intent to persist in science or course grades, but ULTrA significantly correlated with both. We regressed each outcome by ITIS, gender, race, and generation in college. Then, we added ULTrA into the regression model and observed the change in variance predicted (ΔR2). For every construct we measured, adding ULTrA significantly increased the amount of variance predicted.

Significance
The ULTrA survey measures mindset more accurately than the ITIS. Our regression analyses confirmed that the ULTrA survey adds additional, unique predictive power beyond what the ITIS measures. Critically, ITIS was unable to detect relationships that are detected when mindset is measured by the ULTrA Survey, suggesting that previous replication failures may be due, at least in part, to lacking a precise measurement. The ULTrA could potentially be used to explore and mitigate the replication crisis with respect to mindset research in undergraduate education contexts.

Authors

• Lisa B Limeri, Texas Tech University
  Presenting Author

• Nathan Carter
  Non-Presenting Author

• Franchesca Maria Lyra, University of Texas at Austin
  Non-Presenting Author

• Joel Martin
  Non-Presenting Author

• Halle Mastronardo
  Non-Presenting Author

• Jay Patel
  Non-Presenting Author

• Erin L. Dolan, University of Georgia
  Non-Presenting Author