Multiple-Choice Online Causal Comprehension Assessment (MOCCA): Selecting an Item Response Theory

• In Event: First-Year Results for the Multiple-Choice Online Causal Comprehension Assessment: Project MOCCA

Abstract

Objective. This paper reviews the item response theory evidence relating to the pilot MOCCA administration.
Theoretical Framework. Each MOCCA item consists of a story with one sentence missing and four alternative sentences. One alternative is the correct answer, the causally coherent answer. Each incorrect response is one of three types: a paraphrase, a lateral connect, or a local bridging response. Students receive a score indicating their number of correct responses but also three additional scores indicating their number of paraphrase, lateral connect, and local bridging responses.
Because there are multiple forms, all four scores must be equated across forms using item response theory. This paper will describe item response theory models that have been considered and presents preliminary results for one of those models, a multidimensional extension of Bock’s nominal response model that assumes unordered response categories (Bock, 1972).
Method. In the model, the probability that person i will select option o’ on item j is
(1)
where is the response option chosen by person i for item j, is the discrimination parameter for item j and response type o’, is the tendency of person i to select option type o’, and is the location for option type o’ of item j. Whereas Bock’s model is unidimensional, our model contains one dimension for each response type.

Let if person i chooses response type o’ for item j and let if person i chooses an option type other than o’ on item j. Using algebraic manipulations suggested by Tutz (1991), it will be shown that if satisfies Equation 1, then the dichotomous dummy variable for option o’, will have the following form:

(2)
that looks very much like a 2PL model except for the term in the numerator and denominator. This resemblance to the 2PL model has led us to try fitting a 2PL model to each of the four dummy variables , one dummy variable for each response type, to estimate the person parameters for the dimension corresponding to that response type. We are simulating data using Bock’s nominal response model, but estimating the theta parameter for each category using the 2PL model. In most conditions so far, the squared correlations between true and estimated thetas have been above .7 (.55 - .87). Results from the full study will be presented.
For real data collected Spring 2015 from 932 students grades 3 – 5, 1PL and 2PL models were fitted to all response variables for each test form. Sample sizes for each form were small. Marginal reliabilities for the correct answer were all above .9, those for paraphrases responses were all above .7, but those for lateral connect and local bridging were sometimes below .7.
Results and Significance. Real and simulation results suggest that the heuristic approach of using a 2PL model can yield reasonable recovery of person parameters except in extreme conditions. These results are promising not only for progress on MOCCA but also for the development of other assessments that make use of informative distractors.

Authors

• Mark L. Davison, University of Minnesota

• Bowen Liu, University of Minnesota - Twin Cities

• Ben Seipel, California State University, Chico

• Sarah Elizabeth Carlson, University of Oregon

• Gina Biancarosa, University of Oregon