Patterns of Rapid Guessing Behavior Across Items in Low-Stakes Assessments

• In Event: 35.080 - Validity and Fairness Issues in Assessing Higher Education Students

Abstract

It is not unusual for assessments in higher education to be administered under low-stakes conditions in which there are little to no personal consequences associated with an individual examinee’s performance (Cole, Bergin, & Whittaker, 2008). If examinees are not motivated to put forth their best effort, their performance may not be a valid indicator of what they know and are able to do (Wise & DeMars, 2005). Because the validity of the inferences made from low-stakes assessment results is tied to examinees’ effort, it is imperative for test users to measure examinees’ effort and understand its impact on scores (e.g., Standard 11.20; Standards for Psychological and Educational Testing, AERA, APA, NCME, 2000). When assessments are administered using a computer, response time can be used as a measure of examinees’ effort.

Wise and Kong (2005) proposed an index based on response times to differentiate solution behavior on an item from rapid guessing behavior. Although Wise and Kong refer to this index as the solution behavior index, its converse, the rapid guessing (RG) index, is considered here. The RG index has been used in a variety of ways to study examinee motivation. For example, a measure of examinees’ effort based on this index and known as response time effort (RTE; Wise & Kong, 2005) has been used to “filter out” examinees suspected of low-motivation from the data (i.e., motivation filtering ; Sundre & Wise, 2003).

The purpose of the present study is to use latent class analysis to examine patterns of rapid guessing behavior across items. Samples (N1=710; N2=772) of college students were administered two higher education assessments (a 30-item sociocultural knowledge and skills assessment and a 50-item environmental stewardship knowledge and skills assessment) under low-stakes conditions. Their data are being used to answer the following research questions:

1. How many rapid guessing behavior patterns exist and is the number and nature of these patterns the same across assessments?
2. Do examinees with different patterns differ on other variables (e.g., gender, self-reported effort, etc.) in expected ways?
3. Do the patterns align with patterns of low motivated response behavior proposed by other researchers (e.g., Cao & Stokes, 2008; Jin & Wang, 2014; Yamamoto, 1990)?
4. Is there evidence to suggest that motivation filtering based on patterns of rapid guessing behavior is preferable over filtering based on RTE?

There are at least two ways in which the results of this study can be used to address validity issues in low-stakes higher education assessment. First, by understanding the patterns of rapid guessing behavior across an assessment, the results of this study can be used to inform the creation of measurement models for identifying low-motivated examinees and control for their behavior. Second, the results can be used to decide if filtering based on patterns of rapid guessing behavior yields data less contaminated by low motivation and thus is more appropriate for making inferences about college students’ skills and abilities.

Authors

• Dena A. Pastor, James Madison University

• Scott N. Strickman, James Madison University