Growth Mind-Set Versus Not a Fixed Mind-Set: Comparing Positively and Negatively Worded Survey Items

• In Event: 47.090 - Social-Emotional Learning Measurement and Continuous Improvement: Lessons From the CORE Districts

Abstract

Adopting a continuous improvement approach to metric development enables researchers and practitioners to partner in ways that facilitate the development of reliable and valid measures to assess students’ SEL (Davidson et al., 2018). Continuous improvement helps address one of the key challenges facing educators, researchers, and policy makers as they grapple with how to assess social-emotional skills and competencies: deciding the most appropriate scale or instrument to use. Because many of the scales developed by education researchers often rely on relatively small samples (e.g., Blackwell et al., 2007; Duckworth, Tsukayama, & May, 2010), there is often not a strong evidence base supporting the use of certain instruments in the field at scale. In the CORE Districts, district staff and leaders selected four social-emotional competencies to assess: growth mindset, self-efficacy, self-management, and social awareness. As described in Authors (Date), the CORE Districts drew from existing measures that met a series of criteria, including that they were evidence-based, free to administer, practical, parsimonious, and strengths-based.

Subsequent analyses of the first two years of data from the CORE surveys revealed that, compared to the other scales, the growth mindset construct had lower internal consistency as measured by Cronbach’s alpha and higher variance in student responses (Authors, date). Particularly, students in younger grades and students who were not native English speakers were likely confused by the items in the growth mindset scale, potentially because of the negative wording of the items (Authors, date). As a result of these findings, the CORE Districts piloted 10 new growth mindset items in the 2017-18 school year as part of their Innovation Zone initiative. (See http://coredistricts.org/wp-content/uploads/2017/07/CORE.Districts.Innovation.Zone_.QAv2_.pdf for more detail on the Innovation Zone.)

In this paper, we analyze the performance of four of these new items, which are positively worded versions of the original, negatively-worded growth mindset items administered in the CORE Districts from 2015-2017. We repeat prior analyses of the CORE SEL survey measures (Authors, date) to evaluate whether positively worded items have superior psychometric properties compared to the original items. Specifically, we use classical item analysis (including measuring item difficulty, item discrimination, and percent of responses missing), exploratory and confirmatory factor analysis, differential item functioning (DIF) analysis, and item response theory modeling (including estimating nominal response models [NRM; Bock, 1972], partial credit models [PCM; Masters, 1982], and generalized partial credit models [GPCM; Muraki, 1992]).

By comparing the results of this suite of analyses for the original growth mindset items and the revised growth mindset items, we provide empirical evidence that informs the CORE Districts’ decision-making approach for continuously improving their SEL survey measures. In addition, this paper will discuss for the first time results from the innovative approach the CORE Districts are taking to pilot new SEL survey items at a massive scale and review the outcome of this first pilot initiative. Given that the CORE District items are open source and free to administer, this paper stands to make a critical contribution to the field’s growing understanding of appropriate, valid, reliable, and cost-effective approaches for measuring students’ non-cognitive and social-emotional abilities at scale.

Authors

• Caroline Wang

• Libby Pier, Education Analytics

• Robert H. Meyer, Education Analytics

• Daniel M. Bolt, University of Wisconsin - Madison