Search
Program Calendar
Browse By Day
Browse By Time
Browse By Person
Browse By Room
Browse By Unit
Browse By Session Type
Search Tips
Annual Meeting Housing and Travel
Personal Schedule
Sign In
X (Twitter)
THEORETICAL FRAMEWORK
Examinee effort is often a threat to validity, particularly when the assessments are administered in low-stakes conditions where no real consequences are tied to the examinees’ performance (Eklöf, 2010; Wise & DeMars, 2005). When examinees fail to put forth effort, the scores obtained are not representative of their true ability and inferences based on the scores are generally untrustworthy. Several methods have been proposed to increase effort in low-stakes conditions (e.g., Lau, Swerdzeski, Jones, Anderson, & Markle, 2009; Liu, Rios, & Borden, 2015). Of particular interest is effort-monitoring computer-based testing (CBT) introduced by Wise, Bhola, and Yang (2006). When using effort-monitoring CBT, examinees are classified as engaging in solution behavior or rapid-guessing behavior on each item based on their item response times. Examinees engaging in consistent rapid-guessing are shown alerts informing them that their effort appears low and either a) reminding them about the test’s purpose (first alert) or b) warning them that if their effort continues to be of concern, they might be asked to attend a make-up session (second alert). When Wise et al. (2006) administered alerts to examinees who exhibited three consecutive rapid-guesses on an assessment, examinees in the alert condition were found to put forth more effort than examinees in the no-alert condition, which resulted in more valid test scores. Aside from Kong, Wise, Harmes, and Yang (2006), who replicated Wise et al.’s (2006) findings, limited research has been conducted on the effort-monitoring CBT.
STUDY OBJECTIVES
In the current study, we extend Wise et al.’s (2006) study by examining the effect of effort-monitoring CBT on examinee effort when different criteria for displaying alerts are employed. If receiving alerts increases examinee effort, it may be beneficial to display alerts to all examinees, regardless of their test-taking behavior. We aimed to address three primary research questions in our study:
RQ1: What is the effect of effort-monitoring CBT on examinee effort when alerts are displayed to all examinees at fixed points on the assessment?
RQ2: Compared to Wise et al.’s (2006) approach, does displaying alerts at fixed points on the assessment provide similar benefits?
RQ3: Compared to traditional CBT, do examinees exhibit more effort and better performance under effort-monitoring CBT?
METHODS/RESULTS
Data will be collected during the Spring 2018 Assessment Day at a mid-sized university. Examinees will be randomly assigned to one of four alert administration conditions (see Table 1). Examinees’ effort will be measured and compared across conditions using response time effort (RTE) index, which represents the proportion of items for which an examinee engages in solution behaviors. Modified versions of Wise et al.’s (2006) alerts will be used. Examinees in the three effort-monitoring CBT conditions are expected to have higher RTE averages and higher test performance than examinees in the no-alert condition.
SIGNIFICANCE
Pragmatically, if displaying alerts at fixed points on the assessment (e.g., halfway through the assessment), rather than based on examinee test-taking behavior, results in comparable increases in examinee effort, then the former would be less taxing and difficult to implement for practitioners.