A Mixed-Methods Scale Validation for Measuring College Students' Psychological Cost

• In Event: 35.080 - Validity and Fairness Issues in Assessing Higher Education Students

Abstract

The “weed out” culture is prevalent in science, technology, engineering, and mathematics (STEM) disciplines (International Communications Research, 2011). Introductory courses typically have high failure rates where 30-70% of students fail or withdrawal. Unfortunately, students who are “weeded out” of a particular class are often “weeded out” of college altogether (Ramist, 1981). We take a motivational perspective in thinking about the “weed-them-out” culture. Specifically, we set out to understand and measure students’ psychological cost. Cost, a component of Expectancy-Value Theory (EVT; Eccles et al., 1983), can be defined as what has to be sacrificed or suffered as the result of engaging in the activity (e.g., Eccles & Wigfield, 1995). The purpose of this series of studies was to develop a measure of psychological cost for use in understanding college student attrition.

Despite recommendations for strong substantive foundations in scale development (Borsboom & Mellenbergh, 2004; Gehlbach & Brinkworth, 2011; Gorin, 2007; Zumbo & Shear, 2011), the majority of published validity work relies primarily on using factor analysis. We, however, embarked on a series of studies to better understand the substance of cost. We feel substantive development is the most crucial step in understanding what cost is and its importance in higher education. First, we conducted a comprehensive literature review and qualitative study. We used those studies to develop operational definitions and items for expert review. Finally, we tested the psychometric properties of the scale.

From our review, we found evidence of confusion about how to define and measure cost. In each study, cost was operationalized differently, with different items and substantive focus. We synthesized the literature and identified four potential dimensions of cost. To bolster our understanding of the theoretical structure of cost, we conducted focus groups using student participants. We used the focus group data to determine if students experienced cost and if they discussed the dimensions that we uncovered from the literature. We integrated the information from the literature review and qualitative study to converge on an operational definition of cost that contained four dimensions (see Figure 1). We then wrote an initial pool of items.

We recruited eight content experts for a content analysis (Dawis, 1987; Smith & Kendall, 1963). This process provided invaluable feedback regarding clarity and wording. With the revised scale, we collected data from two samples, one for an exploratory factor analysis and one for a confirmatory factor analysis (McCoach, Gable & Madura, 2013). These two samples were used to confirm the hypothesized scale structure. The final scale of 19 items covers the breadth of the construct, while also maintaining the proposed factor structure and excellent internal consistency reliability.

Taken together, we have strong validity evidence for the construct of cost. In the literature and across our studies, we found that cost was salient to students, separate from other motivation constructs, contained multiple dimensions, and is related to student academic outcomes. With a clear operational definition and a comprehensive measurement tool, we are able to deepen our understanding of motivation and persistence among higher education students.

Authors

• Jessica Kay Flake, York University

• Kenn E. Barron, James Madison University

• Chris S. Hulleman, University of Virginia

• D. Betsy Mccoach, University of Connecticut