Multiple-Choice Online Causal Comprehension Assessment (MOCCA) Computer Adaptive Test

• In Event: Updates on Multiple-Choice Online Causal Comprehension Assessment (MOCCA), MOCCA-CAT, and MOCCA-College: A Diagnostic Reading Comprehension Assessment for Struggling Comprehenders

Abstract

The purpose of this paper is to summarize efforts to develop a computer adaptive version of MOCCA, an online reading comprehension assessment for students in grades 3-6. MOCCA has been shown to lead to reliable and valid inferences about both students’ comprehension skills and the cognitive processes they utilize when they do not fully comprehend. However, one potential shortcoming of the assessment is the time required to complete the test, particularly given recent pushback from K-12 teachers, students, and families alike against unnecessary and overly-long testing. In response to these concerns, we have undertaken an effort to make MOCCA computer-adaptive as a means of both reducing testing time and improving the precision of scores. In computer-adaptive testing (CAT), each student experiences a unique assessment in that the items presented are selected based on students’ responses to previous items. In this paper, we share our approach to employing a novel item response theory (IRT) approach to using data from a CAT administration to estimate not only a student’s overall reading comprehension ability, but also the cognitive process propensity they use in the absence of full comprehension. We also share results from our iterative refinement and validation process, including revisions to item format, item specifications, and the number of response options. Results from pilot studies demonstrate that students do not differ in their estimation of how difficult MOCCA is nor in their own performance on MOCCA based on whether the assessment uses CAT or not. We also share early results from our randomized control trial comparing the original, linear MOCCA to the CAT MOCCA with regard to the reliability and precision of scores, as well as test length in terms of minutes taken and the number of items responded to. We conclude by reviewing our next steps in validating MOCCA as a CAT.

Authors

• Patrick Kennedy, University of Oregon
  Presenting Author

• Gina Biancarosa, University of Oregon
  Non-Presenting Author

• Mark L. Davison, University of Minnesota
  Non-Presenting Author

• David J. Weiss, University of Minnesota
  Non-Presenting Author

• Joseph N. DeWeese, University of Minnesota
  Non-Presenting Author

• Ozge Ersan, University of Minnesota-Twin Cities
  Non-Presenting Author

• Tasia Lorissa Brafford, Texas State University
  Non-Presenting Author