Development of a Battery of Diagnostic Reading Comprehension Assessments: Measuring Causal Comprehension

• In Event: Updates on Multiple-Choice Online Causal Comprehension Assessment (MOCCA), MOCCA-CAT, and MOCCA-College: A Diagnostic Reading Comprehension Assessment for Struggling Comprehenders

Abstract

Objective and Theoretical Framework
Reading comprehension is complex and educators find it increasingly difficult to effectively support students on improving comprehension skills. This need is further validated by continued decreases in national reading proficiency for students in Grades 4, 8, and 12 (NAEP, 2019). Students are not a homogenous group of readers and may struggle with reading comprehension in different ways (Brasseur-Hock et al., 2011). Thus, there is a need for better assessments that identify why students struggle with reading comprehension.

MOCCA (e.g., Carlson et al., 2014) identifies how readers struggle with the cognitive processing during reading comprehension. MOCCA mimics the processes found in previous research from struggling comprehenders who either overly paraphrase text, or overly generate connections that elaborate from the text; both failing to fill in causal information (i.e., via causal inferences; McMaster et al., 2012). MOCCA uses a distractor-driven approach (Sadler, 1998) where readers to fill in a missing sentence with one of three sentence types: causally coherent (correct), paraphrase, elaboration.

The following paper presents data from three versions of MOCCA (i.e., Fixed, Computer Adaptive Test or CAT, College). Each incorporates the same theoretical foundation but uses a mixed methods approach in item development. The research question is: To what extent do item changes across three versions of MOCCA provide reliable, valid, and stable scores for identifying comprehension processing for students in Grades 3-6 and postsecondary education?

Methods
Participants took one of the three versions of MOCCA. Demographics for each sample will be provided at the time of the presentation.

Results
Initial evidence of reliability are reported. Additional analyses of validity and stability for each version, and analyses for the MOCCA-CAT, are currently ongoing and will be reported in the presentation.

Table 1
Internal Consistency Reliability (Coefficient Alpha) by Form for Raw Scores on MOCCA-Fixed
N CCI PAR ELA
Form 3.1 481 0.95 0.91 0.82
Form 3.2 501 0.95 0.90 0.81
Form 3.3 463 0.95 0.90 0.80
Form 4.1 414 0.95 0.89 0.83
Form 4.2 416 0.95 0.90 0.82
Form 4.3 403 0.95 0.89 0.82
Form 5.1 357 0.94 0.89 0.82
Form 5.2 346 0.95 0.88 0.85
Form 5.3 343 0.94 0.88 0.83
Note. CCI: Causal Coherent Inference, PAR: Paraphrase, ELA: Elaboration. MOCCA-Fixed participants were recruited from elementary schools across the U.S. (West, Midwest, Northeast, South).

Table 2
Internal Consistency Reliability (Coefficient Alpha) by Form for Raw Scores on MOCCA-College
N CCI PAR ELA
Form 1 596 0.91 0.83 0.77
Form 2 546 0.90 0.81 0.77
Form 3 533 0.91 0.84 0.78
Note. MOCCA-College participants were recruited from postsecondary institutions in three regions across the US (West, Midwest, Southeast).

Conclusions and Significance
Initial findings from the development of different versions of MOCCA have yielded reliability across three forms for readers in Grades 3-5 and postsecondary education (Biancarosa et al., 2016; Davison et al., 2018). MOCCA results provide information about how a reader is doing overall with comprehension processing, and informative information can be used to make instructional decisions to help improve readers’ causal comprehension of text.

Authors

• Sarah E. Carlson, Georgia State University
  Presenting Author

• Gina Biancarosa, University of Oregon
  Non-Presenting Author

• Benjamin Seipel, California State University - Chico
  Non-Presenting Author

• Mark L. Davison, University of Minnesota
  Non-Presenting Author

• Virginia Clinton-Lisell, University of North Dakota
  Non-Presenting Author

• Patrick Kennedy, University of Oregon
  Non-Presenting Author