Text and Item Feature Analysis of the Multiple-Choice Online Causal Comprehension Assessment (MOCCA)

• In Event: First-Year Results for the Multiple-Choice Online Causal Comprehension Assessment: Project MOCCA

Abstract

Objective and Theoretical Framework. The MOCCA is designed to differentiate between two types of poor comprehenders (Authors, 2014) in terms of the types of comprehension processes they use during reading (i.e., overreliance on paraphrasing or extratextual inferences; McMaster et al., 2012). Nonetheless, research has shown that readability, cohesion, and other text features can influence how easy or difficult a text is to comprehend (e.g., McNamara, Graesser, McCarthy, & Cai, 2014). This paper will discuss the textual feature analysis (e.g. Coh-Metrix; McNamara, Louwerse, Cai, & Graesser, 2013), and item feature analysis based on pilot data.
Method. Currently there are 12 MOCCA test forms, four per grade level: 3rd, 4th, and 5th grade. Each MOCCA form consists of 40 items, which use short narrative texts (seven sentences long) written at the respective Flesch-Kincaid Grade Level (Kincaid et al., 1975). Each item is a unique narrative written so that the causal structure of each text (i.e., plot, nature of events) is developed around a main goal that motivates subgoals and events in the text (e.g., Trabasso et al., 1989). Instead of deleting every n words, as seen with traditional cloze tasks (e.g., Deno, 1985), one sentence of each short narrative text is deleted. For each item (i.e., text) the sixth sentence is deleted. Readers must choose among four response types to complete the narrative: (1) a causally coherent inference, (2) a paraphrase, (3) a lateral connection, and (4) a local bridging inference.
To explore and validate the causal structure of each narrative, five versions of all 480 items were processed using Coh-Metrix (McNamara et al., 2013). The five versions included one version in which the 6th sentence remained blank. The remaining four versions included each response (causal connection inference, paraphrase, local bridging inference, and lateral connection inference). We will present results regarding Flesch-Kincaid grade level, cohesion, and coherence.
To explore item features influence on comprehension, we calculated correlations between item difficulty statistics from classical test theory, item response theory results, and item features. Item features included: readability, compound grammatical structures, goal statement complexity and position, the nature of the goal (seeking vs. avoidance), inclusion of a secondary agent, final emotional valence (inferred and type), nature of subgoal, and title information (i.e., character, situation, animal).
Results and Significance. This paper presentation will conclude with a discussion of how these data and results inform on-going item revision and writing. Results inform not only further development of the MOCCA, but also reading test item construction more generally by suggesting which text features are most predictive of difficulty in comprehension.

Authors

• Ben Seipel, California State University, Chico

• Gina Biancarosa, University of Oregon

• Sarah Elizabeth Carlson, University of Oregon

• Mark L. Davison, University of Minnesota

• HyeonJin Yoon, University of Oregon - Center on Teaching and Learning

• Joan Grohman, California State University - Chico