Multiple-Choice Online Causal Comprehension Assessment (MOCCA)–College: Item Response Theory Modeling

• In Event: First-Year Results From the College Version of the Multiple-Choice Online Causal Comprehension Assessment (MOCCA-College)

Abstract

Objective. This paper reviews the item response theory evidence relating to the pilot MOCCA-College administration.
Theoretical Framework. MOCCA-College is composed of narrative and expository text. There is only one item per passage, so there are no dependencies due to a testlet structure. Our model assumes that, despite the two types of text, the items are unidimensional, but an obvious alternative theory is that expository and narrative items define separate dimensions. Unidimensional and two-dimensional structures will be compared. The marginal reliability of comprehension scores will be estimated with item response theory. A bi-polar unidimensional model will be proposed for identifying the two types of poor comprehenders: (1) that is, students who favor paraphrase over lateral connect responses when responding incorrectly vs. (2) those who favor lateral connect responses over paraphrases when responding incorrectly.
Method. Results will be based on responses from 1220 college students who took the assessment in 2019. Students were randomly assigned to one of our 50-item forms, and there was an average of approximately 307 students taking each form. One-and two-dimensional three-parameter logistic models were fitted to the correct responses constraining the difficulty parameter to be equal across items. The model was fitted separately by form. A graded-response model was fitted to a polytomous incorrect response variable: 2 = paraphrase response, 1 = causal coherent response, and 0 = lateral connect response. This response variable is conceived as bi-polar (paraphrase vs. lateral connect) with a correct response conceived as a “neutral” point. A graded response model was fitted to the polytomous variable with the underlying dimension being conceived as a bi-polar dimension: paraphrase vs. lateral connect.
Results and Significance. Here we describe only our early results and their implications.
The one- and two-dimensional models for correct responses have been compared using the likelihood ratio test, AIC, and BIC. The two-dimensional model was a simple structure model with narrative text items along dimension 1 and expository text items along dimension 2. Across all four forms, the estimated correlation between the narrative and expository dimensions was over .95. Fit measures unambiguously favored the unidimensional model for three of the four forms. While there may be some evidence for a second dimension on the fourth form, the high correlation of dimensions on even this form would suggest that the distinction between narrative and expository dimensions is of little practical significance. Results support the use of a single overall achievement score. In the unidimensional models, expository items were more difficult, on average, than narrative items. Narrative items had, on average, higher discrimination parameters. Across forms, marginal reliabilities were in the high .80’s.
The marginal reliability of the polytomous dimension was somewhat disappointing, in the high .60’s. We interpret these results to mean that we cannot report a reliable paraphrase vs. lateral connect score, but we could “flag” poor comprehenders with extreme scores along this dimension as possible paraphrasers or possible lateral connectors with the “flag” having implications for future instruction.

Authors

• Mark L. Davison, University of Minnesota

• Benjamin Seipel, CSU, Chico

• Sarah Elizabeth Carlson, Georgia State University

• Virginia Clinton-Lisell, University of North Dakota

• Patrick Kennedy, University of Oregon

• Bowen Liu, East China Normal University