Measuring Interest: The Open-Ended Response in a Large-Scale Survey

• In Event: Assessment of Interest: New Approaches and New Insights

Abstract

This study was collaboratively designed with faculty members and administrators at nine colleges and universities known to have rigorous undergraduate preparation in STEM. They wanted to know the backgrounds, needs and preferences of the students with whom they work and whether these vary based on gender, URM or first generation status, especially for those who intend to major in STEM but do not. The survey designed targeted five key variables, including interest, due to the collective interests of the stakeholders, which also allows us to study how interest relates to other potentially relevant variables. Items were drawn from existing surveys (e.g., Harackiewicz, et. al., 2000; Midgley, et. al, 2000; Morgan, Issac, & Sansone, 2001; Renninger, et. al., 2011) and/or ethnographic description (e.g. Brainard & Carlin, 1997; Seymour, 1992); they were revised through six focus-groups.
Participants were 4182 (2317 f, 1672 m, 9 other, 184 non-identified) first semester senior undergraduate students. They completed an online questionnaire that included Likert ratings, check boxes, and open-ended response items. Exploratory factor analysis was used to determine the number of factors for each of five variables addressed by the survey: career plans and choices (4), choice of major (4), instructional practices (3), motivational orientation (3), and interests and attitudes (4).
Findings from logistic regression analysis suggest that demographic variables alone are not predictive (regressions with demographic variables and the null model have the same error rates); verifying previous research that the predictive power of demographic variables is small.
Logistic regressions conducted with interest variables are far more predictive. Moreover, the demographic variables become less significant when analyzed with interest. None of the motivational orientation factors predict retention. The error rates are much lower for regressions with the interest variables versus the motivation variables (17.9% compared to 29.9%).
In the open-ended items, respondents described their pre-college major intentions and their explanation of why they did or did not continue with STEM. These data were coded using categorical directed content analysis (Hickey & Kipping, 1996; Hsieh & Shannon, 2005; Potter & Levine-Donnerstein, 1999). This approach involves using existing research and theory to inform data reduction, in addition to allowing identification of emergent categories for coding. The open-ended responses validate the survey responses and suggest eight different student statuses (e.g., those who stay in STEM because of interest or strength in the field, those who stay but struggled or despite some negative experiences).
The largest differences in factor scores across the eight statuses occur from interest factors, choice of major, and instructional practices. These findings are consistent with prior findings on interest and STEM access and retention. Because they were undertaken in relation to a broad set of relevant variables, they allow identification of the links that exist among these variables for students of different statuses—data that have not previously been available. Discussion will focus on the complementarities between the mixed methods employed.

Authors

• K. Ann Renninger, Swarthmore College

• Lynne Steuerle Schofield, Swarthmore College