Development of the Undergraduate Lay Theories of Abilities (ULTrA) Survey

• In Event: Investigating Academic Mind-Sets in New Contexts and With Novel Conceptual Approaches

Abstract

Students’ beliefs about intelligence, called lay theories, affects their motivations, behaviors, and academic success. Research has identified two particularly important lay theories: beliefs about whether intelligence can be improved or is fixed (mindset beliefs; Dweck, 1999) and beliefs about whether everyone, or only some people, have the potential to become highly intelligent (universality beliefs; Rattan et al., 2012). Research suggests that there exists great potential to improve student outcomes by leveraging the positive effects of these lay theories (Yeager et al., 2016). However, research on undergraduates’ students’ lay theories has been mixed. One reason for inconsistent research findings may be that the survey used to measure students’ lay theories does not work well with undergraduates. Our prior research found that the existing mindset survey is problematic because it uses language that undergraduates interpret in varied ways (i.e., “intelligence”; Authors, 2020). Thus, we are developing and validating a new measure of lay theories for STEM undergraduates in accordance with the standards for psychological measurement framework. In this presentation, we will describe the development process and present validity evidence we have collected.
We first sought to identify language about intelligence that undergraduates interpret consistently. We interviewed 45 STEM undergraduates who varied in their personal characteristics and disciplines from 12 institutions about how they conceptualized intelligence and related terms. We identified four candidate terms to replace “intelligence”: analyzing information, applying knowledge, thinking creatively, and learning speed. Based on these results and existing literature on lay theories, we drafted an initial pool of 50 survey items. We then engaged in an iterative process of collecting feedback and revising the item set. One source of feedback was cognitive interviews, in which 29 students read survey items and verbally explain how they interpret the item and their reasoning for their answer. The combination of preliminary interviews to identify suitable language and cognitive interviews provide evidence of response process validity (i.e., that participants interpret and respond to the items consistently). We also collected feedback about the items from experts in lay theories (n=18) and survey design (n=3). Lay theory experts reviewed the items to ensure that they captured the full range of content for mindset beliefs and universality beliefs, and that they did not represent ideas that fell outside of these constructs. This feedback provides evidence of construct validity (i.e., that the items represent the constructs they are intended to measure). Survey design experts reviewed items to ensure that their structure would not introduce confounding sources of variance, such as being negatively worded or double-barreled. We will report lessons learned from each of these sources of data and how the item set has been revised accordingly.
The ultimate research goal is to produce a measure of lay theories suitable for use with STEM undergraduate students. This new measure could be used to answer research questions about lay theories in undergraduates, help instructors learn about the lay theories their students hold and tailor their instruction accordingly, and develop and evaluate interventions to improve academic success.

Authors

• Lisa B Limeri, University of Georgia - Athens

• Nathan Carter

• Franchesca de Lyra

• Halle Mastronardo

• Jay Patel

• Erin L. Dolan, University of Georgia