Developing and Validating Measures of Noncognitive Factors in Middle and High School Students

• In Event: Measuring Motivational Processes and Other Noncognitive Factors: Challenges and Advances in Connecting Research to Practice

Abstract

The goals of this paper are to answer key measurement questions:

• How well can student “noncognitive” factors be measured through self-report surveys?
• How are different noncognitive factors related to student grades?

A large and growing body of research indicates that motivational processes matter for student outcomes. Farrington et al. (2012) reviewed this extensive literature and developed a conceptual model of how these “noncognitive factors”—academic behaviors, academic perseverance, academic mindsets, learning strategies, and social skills—may interact to affect grades. Most existing literature focused on studies of a single noncognitive factor, such as grit (e.g., Duckworth, 2005) or growth mindset (e.g., Dweck, 1986), using survey questionnaires to measure that single factor.

We developed, tested, and implemented a pilot survey of 6th-12th grade students aimed at simultaneously measuring all the factors in Farrington and colleagues’ conceptual model. We also included questions about instruction and other classroom conditions, such as students’ perceptions of their teachers’ classroom practices thought to be important for fostering positive academic mindsets in students. Each noncognitive factor was broken into specific constructs and, where necessary, sub-constructs. For example, Academic Mindsets was broken into nine constructs, including Intrinsic Relevance (the extent to which students found class and classwork to be enjoyable) and Mastery Goal Orientation (the extent to which students wanted to master course material).

For each construct, questions from existing surveys were used, augmented with new items as appropriate to the conceptual model, with some existing items modified for the target population,. Expert review was used to further refine questions. Cognitive interviews were administered to a small number of 6th and 9th grade students to identify issues in the survey response process (Willis, 2005; Tourangeau et al., 2000). All revised questions were asked in a large-scale pilot of 6th-12th grade students (N=8473) called the Becoming Effective Learners Student Survey (BEL-S). The BEL-S survey was administered in Chicago Public Schools and in charter schools around the United States in winter 2013.

Rasch analysis was used to combine items from each construct into a measure. Some measures, such as Intrinsic Relevance, had high reliability and separation, and all items fit within the measure. Other measures, such as Mastery Goal Orientation, had relatively low reliability and separation. In this case, the measure actually included two different factors.

Ordinal logistic regression methods were used to estimate the effect of each noncognitive factor on students’ concurrent grades, controlling for grade level and achievement test scores. Variance was estimated using the Taylor series approximation to control for any clustering at the school level. We will present findings from the Rasch analyses and ordinal logistic regression models, with a specific focus on noncognitive factors related to academic achievement motivation. Discussion will address challenges of measuring noncognitive factors and ways we addressed some of those challenges.

Authors

• Rachel Levenstein, University of Chicago

• Courtney M Thompson

• Camille A. Farrington, University of Chicago