Multiple-Choice Online Causal Comprehension Assessment (MOCCA): Year 2 Classical Test Theory Results

• In Event: 53.077 - Year 2 Results for the Multiple-Choice Online Causal Comprehension Assessment: Project MOCCA

Abstract

Objective and Theoretical Framework. This paper reviews the classical test theory evidence relating to the first national field administration of MOCCA. MOCCA is designed to differentiate between two types of poor comprehenders (Authors, 2014) in terms of the types of comprehension processes they use during reading (i.e., overreliance on paraphrasing or extra-textual inferences; McMaster et al., 2012). Part of the evidence for MOCCA achieving this goal is to examine its internal consistency.

Methods. Over 3700 Grade 3-5 children across 50 schools in 13 states in the USA took MOCCA in spring of 2016. Students were randomly assigned to one of three forms written for their grade level based on the Flesch-Kincaid Grade Level of the MOCCA story with the correct sentence restored (Kincaid, Fishburne, Rogers, & Chissom, 1975). Each MOCCA form consists of 40 items. Students were not required to finish all 40 items (required a minimum of 10), but could review their items if they finished the assessment. Each item had one correct response (causally coherent inference) and two distractors designed to be informative of poor comprehenders’ text processing tendencies: a paraphrase, and a lateral connection. Thus, MOCCA yields three raw scores: one for each answer type. Administration took between 30 and 60 minutes depending on teacher judgment and student pace. Participating districts also provided curriculum-based measurement scores in reading and math, and where applicable, state test scores. These scores will be used to calculate initial validity coefficients for MOCCA.

Results and Significance. Internal consistency coefficients suggest excellent reliability for the causally coherent response type, with Cronbach’s alpha ranging from r = .92 to .94 depending on grade level and form. Coefficients for the paraphrase response type are also good, ranging from r = .88 to .86 depending on the grade level and form. Coefficients for the lateral connection are improved from previous administrations, ranging from r = .71 to .81 depending on the grade level and form (previous ranges: r = .50-.86). Validity analyses are ongoing.

Results suggest that from a classical test theory perspective, MOCCA can successfully distinguish good comprehenders from poor comprehenders and identify paraphrasers, and preliminarily identify lateral connectors. When reviewed in reference to comprehension theory, item-level results (i.e., correlations between the score for each item and the total score for each response type) can inform item and response type revisions. Overall, results support the idea that a multiple-choice assessment with informative distractors can be used to give information on not only whether students comprehend, but also how they comprehend.

Authors

• Gina Biancarosa, University of Oregon

• Mark L. Davison, University of Minnesota

• Ben Seipel, University of Wisconsin - River Falls

• Sarah Elizabeth Carlson, University of Oregon

• HyeonJin Yoon, University of Nebraska-Lincoln