Assessing Survey Satisficing: The Impact of Unmotivated Questionnaire Respondents on Data Quality

• In Event: Causes and Consequences of Student Self-Reported Social-Emotional Learning: Lessons From the CORE Districts

Abstract

As U.S. policymakers increasingly incorporate social-emotional learning (SEL) into state and federal policies (CASEL, 2019), researchers face a growing need to use student survey data to inform decision-making. This need raises concerns around data quality and whether respondent motivation (or lack thereof) might cause sub-optimal responding (i.e., satisficing) to such an extent that student surveys are not viable sources of information. We examine the pervasiveness and impact of student satisficing in a large-scale SEL survey to inform this potential issue.

Researchers have adapted the concept of satisficing—the act of conserving mental energy in decision-making (Simon, 1957)—to explain sub-optimal survey responses (Krosnick, 1991; Tourangeau, 1984). Strategies for detecting satisficing include various methods (Barge & Gehlbach; Steedle et al., 2019) that range in complexity and applicability for different survey contexts. In this study, we provide straightforward, accessible strategies for systematically defining, calculating, and reporting satisficing in large-scale datasets that should work for almost all survey contexts.

We examined secondary data collected through a policy research center. The sample included 409,721 students from five California districts (146,126 elementary; 125,747 middle; and 137,838 high school students). Analyses relied on students’ responses to a 25-item SEL survey comprised of four scales: growth mindset, regulation, self-efficacy, and social awareness.

Based on exploratory analyses, we pre-registered a set of hypotheses (Authors, 2019) according to recommended practices (Gehlbach & Robinson, 2018). We defined satisficing as engaging in at least one of three response patterns: early termination, failure to complete the full survey; non-response, missed items; and straight-line responding, selecting the same response option at least ten items in a row.

Students in our sample satisficed extensively; 33.74% engaged in at least one type of satisficing. Of the three types, straight-lining impacted the greatest number of survey items (approximately 15) compared to non-response and early termination (1.77 and 3.52, respectively). Moreover, because students overwhelmingly selected the most extreme right-hand response option when straight-lining (M=84%), the strategy also impacted students’ mean scores. The full sample had higher means for regulation, self-efficacy, and social awareness than the high-fidelity sample (i.e., non-straight-liners; Cohen’s d=.08, .10, and .07, respectively). After correcting for reverse-scored items, the full sample had a lower growth mindset mean than the high-fidelity sample (Cohen’s d=-.07).

Despite the prevalence of student satisficing, its impact on data quality appeared surprisingly small (as evidenced by low effect sizes). Of course, the magnitude of effect sizes ranges across research contexts—what may be a small effect size in one domain may represent meaningful change in others. Thus, we recommend that researchers conduct analyses with and without satisficers to confirm that the practice does not meaningfully impact interpretation of their findings. As schools and districts increasingly use student survey measures to inform policy decisions, we aimed to support researchers by providing a foundation for defining and calculating satisficing in large-scale datasets.

Authors

• Christine Calderon Vriesema, University of Wisconsin - Eau Claire

• Hunter Gehlbach, Johns Hopkins University