Automated Scoring in Complex Tasks Through Multidimensional Item Response Theory–Bayes (mIRT-bayes), a Hybridized Measurement Model

• In Event: 66.030 - The State of the Art in Automated Scoring of Science Inquiry Tasks

Abstract

Complex technology-enhanced tasks are rapidly emerging in educational measurement. Often called virtual performance tasks, they can offer assessment evidence that includes questions and answers, but they can also provide other streams of construct-salient information. In a digital world, this may include interactions, choices, behaviors, processes, reasoning facets, information foraging, question posing and other evidence that schools, teachers, policy makers and other educational stakeholders find helpful for engaging and authentic assessment. If interpretable, scorable and shown to be of high quality and salient to the construct, such evidence may prove valuable in large-scale assessments.
Here, a two-stage mIRT-bayes modeling approach is presented as a hybrid avenue for automated scoring in virtual performance tasks in science assessment. Using a two-stage approach, simple observables that cannot readily provide scores themselves (semi-amorphous data) are conceived as clusters salient to the construct and designed into the task to be accumulated as a pattern of scores. This paper will show how each cluster is accumulated and automatically provided with a conceptually derived and empirically supported score through a Bayes Net approach. These scores are then entered into a mIRT engine to estimate overall proficiency over a set of scores, for the full mIRT-bayes model.
Applied to simulation-based data from Harvard’s Virtual Performance Assessments as the data source, results show reduction of the standard error of measurement as compared to use of mIRT alone without the accumulation of the semi-amorphous salient Bayes Net clusters. Overall for respondents, a finer grain-size of inference is made possible using the automated scoring hybridization without additional testing time. Furthermore, the semi-amorphous clusters reveal properties, in this case, of maximum item discrimination in the scoring set, as compared to other selected and constructed response items also designed as salient to the construct. The results therefore suggest the perspective of hybridizing models (mIRT-bayes) and including clusters of semi-amorphous data where salient. Drawbacks here include the limited data sets (two data sets of about 1,800 cases each), the exploration of only two tasks, and the inherent “chunkiness” of performance-based tasks generally, whether administered digitally or hands-on.
This paper motivates how mIRT-bayes can potentially be useful by retaining the two-stage estimation and conceptualizing the first stage as an automated scoring engine. While hybridized measurement models can potentially be estimated in a single stage, modularity here has advantages. Modularity in both evidence identification and evidence accumulation are very convenient options for both building and using psychometric machinery in complex assessments such as games and simulations. Being able to assemble re-usable fragments of Bayes nets that correspond to small chunks of relationships believed to be understood by learning scientists is useful. That these can be designed to suit situations as they arise is precisely one of the main reasons that Bayes nets are promising for inference in dynamic situations. However, retaining the overarching mIRT model to accumulate the small clusters into larger proficiency estimates hybridizes the measurement model. This can help to substantiate the evidence model with mature measurement machinery for use in innovative assessments.

Author

• Kathleen Scalise, ETS