Does It Matter When Examinee Motivation Is Measured? A Moderated Mediation Analysis

• In Event: Student Motivation and Reactions During Low-Stakes Tests for Institutional Accountability: Issues and Potential Solutions

Abstract

THEORETICAL FRAMEWORK
Test scores collected in low-stakes contexts may reflect examinee ability and motivation (Wise & Smith, 2016). Expectancy-value theory applied to examinee motivation suggests examinees’ perceived value of a test indirectly affects test performance via examinee effort. This empirically supported indirect effect, however, is often modeled using importance and effort scores measured after test completion, which does not align with their theoretically specified temporal order. Importance and effort scores collected after the test may reflect self-protective attributions of performance (Weiner, 1985). However, given students tend to exhibit behavior in accordance with their behavioral intentions, asking examinees to rate their importance and effort before test completion may have the benefit of behavioral commitment (Carver & Scheier, 2000).

PURPOSE
Using an institutional accountability testing program, we examined the impact of timing of measurement of importance and effort, relative to test completion. The purpose of this study was to examine (1) the impact of behavioral commitment on the indirect effect of retrospective importance on performance via retrospective effort; (2) if the indirect effect differs depending on if examinees report importance and effort prospectively versus retrospectively; and (3) if average test performance differs as a function of timing of measurement (i.e., behavioral commitment).

METHODS
Data were collected from a mid-sized public university (N=1,145). First-year college students were randomly assigned to one of three conditions, where (1) importance and effort were measured retrospectively; (2) importance was measured prospectively; and (3) importance and effort were measured prospectively (see Figure 1). Additionally, importance and effort were measured retrospectively in conditions two and three.

RESULTS
When modeling retrospective importance and effort, moderated mediation analyses indicated the practically significant indirect effect was invariant across conditions. When modeling prospective and retrospective importance and effort, the unstandardized indirect effect was invariant across conditions. However, given the relatively smaller variance in prospective effort, the standardized indirect effect was smaller when modeling prospective scores than retrospective scores. Most importantly, examinees who prospectively reported intended effort had significantly higher test performance than examinees who did not prospectively report effort.

CONCLUSIONS/IMPLICATIONS
Invariance of the unstandardized and standardized indirect effects when modeling retrospective importance and effort indicated no effect of behavioral commitment on retrospective importance and effort. Thus, behavioral priming may not be an effective motivation intervention for increasing test performance via increasing retrospectively reported test importance and expended effort. However, priming students by asking them to report intended effort before the test had a small, but arguably practically significant impact on test performance. Thus, priming examinees is affecting performance via some mechanism (e.g., mediator) other than importance and effort and may be a cheap and moderately effective method of increasing test performance.

Authors

• Aaron Myers, James Madison University

• Sara J. Finney, James Madison University