Multiple-Choice Online Causal Comprehension Assessment (MOCCA): Classical Test Theory Results

• In Event: First-Year Results for the Multiple-Choice Online Causal Comprehension Assessment: Project MOCCA

Abstract

Objective. This paper reviews the classical test theory evidence relating to the pilot MOCCA administration.
Theoretical Framework. MOCCA is designed to differentiate between two types of poor comprehenders (Authors, 2014) in terms of the types of comprehension processes they use during reading (i.e., overreliance on paraphrasing or extra-textual inferences; McMaster et al., 2012). Part of the evidence for MOCCA achieving this goal is to examine its internal consistency.
Methods. Over 900 Grade 3-5 children in California and Oregon took the MOCCA in spring of 2015. Students were randomly assigned to one of four forms written for their grade level based on the Flesch-Kincaid Grade Level of the MOCCA story with the correct sentence restored (Kincaid, Fishburne, Rogers, & Chissom, 1975). Each MOCCA form consists of 40 items, but students were allowed to skip items and did not need to finish the assessment. Each item had one correct response (the causally coherent inference) and three distractors designed to be informative of poor comprehenders’ text processing tendencies: a paraphrase, a lateral connection, and a local bridging inference. Thus, MOCCA yields four raw scores: one for each answer type. Administration took between 30 and 60 minutes depending on teacher judgment and student pace. Participating districts also provided curriculum-based measurement scores in reading and math and where applicable state test scores. These scores will be used to calculate initial validity coefficients for MOCCA.
Results. Internal consistency coefficients suggest excellent reliability for the causally coherent response type, with Cronbach’s alpha ranging from .916 to .964 depending on grade level and form. Coefficients for the paraphrase response type are also good, ranging from .713 to .894. Coefficients for the lateral connection (and local bridging) are less satisfactory, ranging from .494 to .863. Validity analyses are ongoing.
Significance. The results suggest that from a classical test theory perspective, MOCCA can successfully distinguish good comprehenders from poor comprehenders and identify paraphrasers, but that additional development work may be needed to more reliably identify lateral connectors. When reviewed in reference to comprehension theory, item-level results (i.e., correlations between the score for each item and the total score for each response type) can inform item and response type revisions. Overall, results support the idea that a multiple-choice assessment with informative distractors can be used to give information on not only whether students comprehend, but also how they comprehend.

Authors

• Gina Biancarosa, University of Oregon

• Mark L. Davison, University of Minnesota

• Bowen Liu, University of Minnesota - Twin Cities

• Sarah Elizabeth Carlson, University of Oregon

• Ben Seipel, California State University, Chico

• HyeonJin Yoon, University of Oregon - Center on Teaching and Learning