Replication, Attenuation, and Null Effects of Value-Reappraisal and Comparison Interventions

• In Event: 30.032 - Opening the File Drawer for Innovation in Task Value Intervention

Abstract

Research on motivational and social-psychological interventions has suggested positive effects on students’ attitudes, behavior, and academic achievement from fairly brief exposure to interventions (see Karabenick & Urdan, 2014; Yeager & Walton, 2011). Moreover, research has found that students with at-risk characteristics such as low self-efficacy (Hulleman & Harackiewicz, 2009) and low academic performance (Paunesku, et al., 2015), and students of underrepresented groups (Harackiewicz, et al., 2013), may be more likely to benefit from motivational and social-psychological interventions. In order to more fully evaluate the significance and reproducibility of intervention effects, it is critical to conduct replication studies and learn from findings that suggest null, attenuating, or inconsistent effects. In addition, it is important to identify and utilize methodological approaches that could help protect against the reporting of spurious results. Furthermore, the null effects of comparison interventions should be used to inform the literature from which they were derived.
This presentation will focus on value-reappraisal interventions and two comparison conditions, goal-setting and learning-strategies interventions. Synthesizing results across four studies, we will discuss lessons learned from null and/or attenuating intervention effects; the degree to which intervention effects have been found to replicate; methods to protect against the inflation of Type I Error; and methods to examine the sensitivity of intervention effects to the inclusion of covariates, decisions regarding missing data, and study-inclusion criteria.
All four studies used random assignment to groups and examined the effects of value-reappraisal interventions on course-performance outcomes and self-report measures of expectancies and values. Studies 1 (N=82) and 2 (N=88) were conducted in introductory statistics courses. Study 1 (Acee & Weinstein, 2010) compared a value-reappraisal intervention to a control condition. Study 2 (Acee, 2009) compared a value-reappraisal intervention to a goal-setting comparison condition and a control condition.
Studies 3 (N=342) and 4 (N=349) were conducted in introductory physics courses for non-STEM majors. Study 3 (Acee, et al., 2016) compared two types of value-reappraisal interventions (writing activities vs. messages and writing activities) to a learning-strategies comparison condition. Study 4 compared two types of value-reappraisal interventions (messages vs. messages and writing activities) to a learning-strategies comparison condition.
In studies 1 and 2, intervention effects on endogenous instrumentality partially attenuated over the two weeks following the intervention. This finding led us to include three intervention doses in studies 3 and 4, and attenuation was not observed. Null intervention effects on intrinsic value in studies 1 and 2 influenced us to not target intrinsic value directly in future interventions. Mixed results of studies 1 and 2 on course-performance outcomes prompted us to conduct studies with stronger power to examine intervention interactions with students’ baseline expectancies, values, and prior achievement. Studies 3 and 4 indicated intervention interactions on course-performance outcomes but some of these interactions had inconsistencies and sensitivities to analytical approaches. Null effects of comparison interventions were consistent across studies. Our synthesis of lessons learned from null and attenuating effects of value-reappraisal and comparison interventions could help inform future interventions and protect against the file-drawer problem.

Authors

• Taylor Wayne Acee, Texas State University-San Marcos

• Theresa V. Hoang, Texas State University

• Darolyn A. Flaggs, Texas State University