Multiple-Choice Online Causal Comprehension Assessment (MOCCA) Item Development

• In Event: First-Year Results for the Multiple-Choice Online Causal Comprehension Assessment: Project MOCCA

Abstract

Objective. This paper will discuss the original item development for the Multiple-choice Online Causal Comprehension Assessment (MOCCA). It includes details on the design and results from an extensive review of all items by teachers serving Grades 3 to 5 students.
Theoretical Framework. MOCCA is designed to differentiate between two types of poor comprehenders (Authors, 2014) in terms of the types of comprehension processes they use during reading (i.e., overreliance on paraphrasing or extra-textual inferences; McMaster et al., 2012). They use other comprehension processes that are strategic and helpful, but that fail to maintain the causal continuity of the text. Specifically, poor comprehenders tend to paraphrase (i.e., repeat or rephrase prior text information) or to make lateral connections (i.e., elaborative inferences or personal associations). These trends have been found repeatedly with intermediate grade readers (e.g., Authors, 2014; McMaster et al., 2012; Rapp et al., 2007).
Item Development Methods. Each MOCCA item is a seven-sentence narrative written so that the causal structure of each text (i.e., plot) is developed around a main goal that motivates subgoals and events in the text (e.g., Trabasso, van den Broek, & Suh, 1989). For each item (i.e., text) the sixth sentence is deleted. Readers must choose among four response types to complete the narrative. The four responses are: (1) a causally coherent inference (the deleted sentence and best response), (2) a paraphrase, (3) a lateral connection, and (4) a local bridging inference. Causally coherent inferences are the only response that completes the story in a causally coherent manner. Paraphrases are always incorrect and mimic what one group of poor comprehenders tend to do. Lateral connections are also incorrect, but mimic what another group of poor comprehenders tend to do. Finally, local bridging inferences are also incorrect, but do not tend to distinguish between types of comprehenders. After iterative development of item writing specifications, over 500 items were written by a team of seven writers that included the project principal investigators. Each item was reviewed and revised by two or more members of the item writing team (not including the original item writer), at which point the informative distractors were written and the items were subjected to another round of review and revision.
Data Sources. Each item was reviewed by a teacher review panel consisting of four Grade 3-5 teachers, one special education teacher, and a Title 1 room teacher who worked in a Spanish-English dual language school and also had extensive English learner experience, including personal experience. Items teachers flagged as problematic underwent further review and revision as needed before the test forms for the pilot administration were created.
Results and Significance. Teachers flagged items as needing revision for many reasons, but the most consistent was because a story did not “need” the sixth sentence. We discovered in many of these cases the teachers had inferred the sixth sentence without realizing they had done so. Results suggest teachers may not always be aware of the inferences readers need to make to maintain causal coherence when reading.

Authors

• HyeonJin Yoon, University of Oregon - Center on Teaching and Learning

• Gina Biancarosa, University of Oregon

• Sarah Elizabeth Carlson, University of Oregon

• Mark L. Davison, University of Minnesota

• Ben Seipel, California State University, Chico

• Joan Grohman, California State University - Chico