Multiple-Choice Online Causal Comprehension Assessment (MOCCA)–College Item Development

• In Event: First-Year Results From the College Version of the Multiple-Choice Online Causal Comprehension Assessment (MOCCA-College)

Abstract

Objective. This paper will discuss item development for the Multiple-choice Online Causal Comprehension Assessment-College (MOCCA-College). It includes details on the design and results from an extensive review of all items from the original MOCCA and the need for expository texts at the collegiate level.
Theoretical Framework. MOCCA-College is designed to differentiate between two types of poor comprehenders in terms of the types of comprehension processes they use during reading (i.e., overreliance on paraphrasing or extra-textual inferences; McMaster et al., 2012, Authors, 2014). These readers use other comprehension processes that are strategic and helpful, but that fail to maintain the causal continuity of the text. Specifically, poor comprehenders tend to paraphrase (i.e., repeat or rephrase prior text information) or to make lateral connections (i.e., elaborative inferences or personal associations).
Item Development Methods. Each MOCCA-College item is a seven-sentence text written so that the causal structure of each text is developed around a main goal or idea that motivates subgoals or events in the text (e.g., Trabasso, van den Broek, & Suh, 1989). In each item the sixth sentence is deleted. Readers must choose among three response types to complete the text: (1) a causally coherent inference (the deleted sentence and best response), (2) a paraphrase, and (3) a lateral connection. Causally coherent inferences are the only response that completes the text in a causally coherent or cohesive manner. Other responses leave a gap of information; the text is incomplete and not cohesive. Paraphrases are always incorrect and mimic what one group of poor comprehenders tend to do. Paraphrases either summarize the main idea or restate the goal of the main character. Lateral connections are also incorrect, but mimic what another group of poor comprehenders tend to do. Lateral connections contain elaborations or asides of core text without completing the causal gap. After iterative development of item writing specifications, over 250 items were written or revised by a team of four writers that included the project principal investigators. All items and distractors were subjected to a several rounds of review and revision by the item-writing team and an independent content review panel consisting of teachers/instructors.
Data Sources. The content review panel consisted of high school college-prep English teachers, college Freshman English professors, and student support personnel—all individuals who work with the target population. Items were evaluated on a variety of criteria including: cohesion, vocabulary, bias (privilege and offensive), background knowledge, accuracy, engagement, appropriateness, content, syntax, and length. Items that were flagged as problematic underwent further review and revision as needed before the test forms for the pilot administration were created.
Results and Significance. The content review panel flagged items as needing revision for several reasons, but the most consistent was because a story did not “need” the sixth sentence. We discovered in many of these cases the reviewers had inferred the sixth sentence without realizing they had done so. Results suggest teachers/instructors may not always be aware of the inferences readers need to make to maintain causal coherence when reading.

Authors

• Benjamin Seipel, CSU, Chico

• Sarah Elizabeth Carlson, Georgia State University

• Mark L. Davison, University of Minnesota

• Virginia Clinton-Lisell, University of North Dakota

• Patrick Kennedy, University of Oregon