Text and Item Feature Analysis of Multiple-Choice Online Causal Comprehension Assessment (MOCCA): Year 2 MOCCA Results

• In Event: 53.077 - Year 2 Results for the Multiple-Choice Online Causal Comprehension Assessment: Project MOCCA

Abstract

Objective and Theoretical Framework. Research has shown that readability, cohesion, and other text features can influence how easy or difficult a text is to comprehend (e.g., McNamara, Graesser, McCarthy, & Cai, 2014). The MOCCA, a new diagnostic reading comprehension assessment, has been designed to differentiate between two types of poor comprehenders (Authors, 2014) in terms of the types of comprehension processes they use during reading (i.e., overreliance on paraphrasing or extratextual inferences) (McMaster et al., 2012). This paper discusses the efforts made to examine the textual features (e.g. Coh-Metrix; McNamara, Louwerse, Cai, & Graesser, 2013), and item features based on Year 2 data from Project MOCCA (i.e., revised stories from Year 1 data).

Methods. Currently, there are 9 MOCCA test forms, three per Grades 3-5. Each MOCCA form consists of 40 items, which includes short narrative texts (seven sentences long) written at the respective Flesch-Kincaid Grade Level (Kincaid et al., 1975). Each item is a unique narrative written so that the causal structure of each text (i.e., plot, nature of events) is developed around a main goal that motivates subgoals and events in the text (e.g., Trabasso et al., 1989). Instead of deleting every n words, as seen with traditional cloze tasks (e.g., Deno, 1985), one sentence of each short narrative text is deleted. For each item (i.e., text) the sixth sentence is deleted. Readers must choose among three response types to complete the narrative: (1) a causally coherent inference, (2) a paraphrase, and (3) a lateral connection.

To explore and validate the causal structure of each narrative, four versions of the 360 items were processed using Coh-Metrix (McNamara et al., 2013). The four versions included one version in which the 6th sentence remained blank. The remaining three versions included each response (causally coherent inference, paraphrase, lateral connection).

Results and Significance. To explore how text features influence comprehension, we calculated correlations between item difficulty statistics from classical test theory, item response theory, and item features. Item features assessed through Coh-Metrix and human-coding included: readability, compound grammatical structures, goal statement complexity and position, goal attainment, inclusion of a secondary agent, final emotional valence (inferred and type), nature of subgoal, title information (i.e., character, situation, animal), and the causal cohesion gap. Results are discussed in terms of how item features continue to inform on-going item and response type revision. In addition, we discuss how the development of MOCCA as a diagnostic reading comprehension test can inform item construction more generally by suggesting which text features are most predictive of difficulty in comprehension.

Authors

• HyeonJin Yoon, University of Nebraska-Lincoln

• Gina Biancarosa

• Sarah Elizabeth Carlson, University of Oregon

• Ben Seipel, University of Wisconsin - River Falls

• Mark L. Davison, University of Minnesota