Validation Study of MOCCA-College

• In Event: Updates on Multiple-Choice Online Causal Comprehension Assessment (MOCCA), MOCCA-CAT, and MOCCA-College: A Diagnostic Reading Comprehension Assessment for Struggling Comprehenders

Abstract

Objective
The purpose of this paper present results from a national validation study of MOCCA-College. MOCCA-College is a diagnostic reading comprehension assessment.
Theoretical Framework
Understanding text requires the coordination of multiple cognitive processes by a reader. Decades of research has provided much awareness about those processes (e.g., strategies, thoughts) which determine how and why some readers struggle with reading comprehension. Yet, traditional reading comprehension assessments generally do not assess the underlying processes that can cause comprehension to falter (Magliano, Millis, Ozuru, & McNamara, 2007), but rather focus on the result (i.e., product) of reading.
There has been recent and continuing pressure to develop measures to identify, diagnose, and address the reading comprehension processing needs of students in postsecondary institutions (e.g., Holschuh & Paulson, 2013; Koon & Petscher, 2016). MOCCA-College addresses these needs by functioning as a diagnostic tool of specific cognitive processes used by college students who struggle with reading comprehension.
By design, MOCCA-College is diagnostic of student comprehension errors based on known cognitive patterns of poor comprehenders (Authors, 2014, McMaster et al., 2012). MOCCA-College is a sentence-level maze task with diagnostic distractors. Each test item consists of a paragraph with a sentence missing. From three alternatives, the student must select the sentence that best completes the text. MOCCA-College items have three types of responses, one correct response (i.e., a causal coherent inference) and two types of incorrect responses that mimic a common cognitive processes seen from poor comprehenders (i.e., paraphrases, elaborations). MOCCA-College uses both expository and narrative texts appropriate in content and readability for college students. MOCCA-College has three forms with 40 items each. There are 10 anchor items used across all three forms.
Methods & Data Sources
College students (N=1,685) from across the United States participated in the study and completed one the three forms (C1 = 596, C2 = 546, C3 = 533). Of those, approx. 1500 completed a voluntary demographic survey. Full demographics will be presented in session.
Results and Significance
Across the three forms, the means for each response type were similar. For the number correct. On average, students answered about 75% of the items correctly.

The internal consistency reliability (alpha) for each of the raw scores was excellent. The
reliabilities for the number correct scores are all at or slightly above .90. Those for the
paraphrase response are all above .80 (great). Those for the elaboration response are all above .75 (good). The pattern is the same as that found in earlier studies of the MOCCA test for elementary school students: the highest reliabilities for the correct score followed by the paraphrase score (Authors, 2019).

MOCCA-College results were strongly correlated with construct validity assessments than with discriminant validity assessments (e.g., ACT Reading r = .40; ACT Math r = .33; ACT Science r =.32).

These preliminary results show that MOCCA-College has potential to identify struggling comprehenders and diagnose why they struggle.

Authors

• Benjamin Seipel, California State University - Chico
  Presenting Author

• Sarah E. Carlson, Georgia State University
  Non-Presenting Author

• Mark L. Davison, University of Minnesota
  Non-Presenting Author

• Virginia Clinton-Lisell, University of North Dakota
  Non-Presenting Author

• Patrick Kennedy, University of Oregon
  Non-Presenting Author