Improving Test-Taking Effort in Low-Stakes Group-Based Educational Testing: A Meta-Analysis of Interventions

• In Event: Student Motivation and Reactions During Low-Stakes Tests for Institutional Accountability: Issues and Potential Solutions

Abstract

STUDY OBJECTIVES
Four decades of research have shown that students’ low test-taking effort is a serious threat to the validity of score-based inferences for low-stakes, group-based educational assessments. Although researchers have attempted to improve test-taking effort via interventions, results have been inconclusive. To address this limitation, a meta-analysis was conducted to identify which intervention types work best.

THEORETICAL FRAMEWORK
Four broad categories of strategies have been proposed to improve students’ test-taking effort: (a) increasing test relevance, (b) modifying assessment design, (c) promising feedback, and (d) offering external incentives (Wise & DeMars, 2005). However, it is unknown how many studies have implemented test-taking effort interventions, the mechanisms employed to manipulate test-taking effort, their overall effect, or the role that contextual variables play in impacting study outcomes. To address these limitations, the following research questions were evaluated:
1. What is the impact of interventions on improving test-taking effort and performance on low-stakes assessments?
2. What are the contextual variables (e.g., participant, methodological, and assessment characteristics) that moderate the impact of such interventions?
3. Which intervention type is most effective in improving test-taking effort and/or performance?

METHODS
Four distinctive search strategies were employed to identify appropriate literature: (a) a reference database search, (b) internet browsing, (c) expert consultation, and (d) citation searches (both backward and forward). Studies were only included if researchers: (a) used a treatment-control group design, (b) administered a low-stakes educational assessment, and (c) evaluated test-taking effort and/or test performance as outcomes. Eight moderator variables were coded: (a) participant age, (b) participant gender, (c) experimental design, (d) participant recruitment strategy, (e) performance measure item type, (f) length of performance measure, (g) publication type, and (h) intervention type (inter-rater reliability was found to be adequate). To examine the impact of moderator variables on effect size estimates, a random-effects meta-regression model was evaluated for two dependent variables: (a) test-taking effort, and (b) test performance.

RESULTS
The analysis included 45 studies (N = 15,962) that produced 44 and 87 effect sizes of test-taking effort and test performance, respectively. Overall, interventions appeared to improve test-taking effort and test performance by 0.20 and 0.13 SD, respectively. Although test performance effect sizes were found to be robust across all moderator variables analyzed, test-taking effort was moderated by participant recruitment strategy and publication type. Specifically, effect size increases of .30 SD were found for studies that: (a) did not mandate student participation, and (b) published in journals. After controlling for these moderators, the largest effects on test-taking effort were obtained by the external incentives (0.52 SD) and assessment design intervention types (0.42 SD); however, the latter did not lead to significant improvements in test performance. In analyzing the impact of external incentives on test performance, we found that providing nonmonetary incentives was found to lead to an improvement of 0.23 SD.

SIGNIFICANCE
The findings from this study have direct implications for how institutions should sample and employ motivation interventions to improve test-taking effort, and thus the validity of score-based inferences, made from low-stakes educational assessments.

Authors

• Joseph A. Rios, Educational Testing Service

• Ou Lydia Liu, ETS