Are Test-Takers' Reflection on Test-Taking Processes Predictive of Their Performance on a Listening Test?

• In Event: Using Mixed Models to Investigate Tendency, Process, and Typology in Item Responding

Abstract

1. Purpose
The primary goal of this study is to investigate whether test-takers’ self-reflection of test-taking processes are predictive of their item-level performance. The secondary purpose is to demonstrate the use of mixed logistic regression to model item-specific and person-specific effects and handle observation dependence in the item response data.
2. Perspective
The Standards for Educational and Psychological Testing (AERA, APA, & NCME, 2014) presents evidence based on response process as one of the five major sources of validity evidence. Most previous research on response processes focuses on cognitive factors. In this study, we take a broader view of response process proposed by Hubley and Zumbo (2017), which embraces test-takers’ behavior and affection besides cognition.
3. Methods
Participants. Data are collected from 189 adults (66.3% were females) with a wide range of listening proficiency. The average age was 35.9 (ranging from 18 to 70).
Measures. CELPIP-General Listening. Our investigation included two listening comprehension tasks from the Canadian English Language Proficiency Index Program-General (CELPIP-General). Each task comprised of an audio passage and a list of multiple-choice questions. All test-takers answered the 11 questions from the two tasks.
Self-reflection Questionnaire of Response Processes. The questionnaire consisted of ten statements, each describing a mental activity in solving the task (see Table 1). Immediately after answering all questions in each task, test-takers rated the extent to which these statements reflected their response processes on a 5-point Likert-type scale from never (0) to always (4). The ten statements are broadly categorized into construct-relevant and construct-irrelevant variables depending on their relevance to comprehending meaning. Construct-relevant processes are directly engaging in comprehending meaning (e.g., understanding main ideas), while construct-irrelevant ones are not (e.g., guessing blindly).
Analysis. We chose the mixed logistic regression to model the relationship between the binary item response outcomes and the response process variables. Each of the ten response process variables was tested separately to investigate its effect (slope) on test-takers’ item-level performance. The specification of random slopes across test-takers allowed the prediction to be person specific. Meanwhile, the specification of cross-item and cross-person random intercepts allowed the item difficulties and test-taker abilities to be accounted for (Baayen, Davidson, & Bates, 2008).
4. Results
The results show that self-reported high levels of construct-irrelevant response processes (R4, R5, R6, R7, and R10 in Table 1) were significantly associated with poorer item performance. In contrast, self-reported high levels of construct-relevant response processes (R1, R2, R3, R8, and R9 in Table 1) were not predictive of item performances. The results also suggest that the chance of answering a test item correctly by guessing varied among test-takers, after taking into account item difficulty and test-takers’ listening proficiency.
5. Significance
Scholars in language testing have called for the inclusion of test-taking processes as part of construct validation (Anderson, Bachman, Perkins, & Cohen, 1991). Despite its importance, such research is scarce. Specifically, little work exists in the listening comprehension testing literature. Our study fills this gap in validation research by examining the link between response processes and item performance.

Authors

• Michelle Yue Chen, Paragon Testing Enterprises / The University of British Columbia

• Amery D. Wu, University of British Columbia

• Yan Liu, The University of British Columbia