Scoring and Analyzing a Collaborative Problem Solving Assessment for NAEP

• In Event: 53.033 - Investigating the Inclusion of Collaborative Problem Solving in the Nation's Report Card

Abstract

Students in collaborative settings talk, negotiate, hypothesize, revise and respond, orally, with gestures, and on line with chats and emoticons, acronyms, and so on. All of these activities seem to matter, within the context of collaboration. Data from Collaborative Problem Solving (CPS) tasks can be characterized as individual and team (collective) outcome data, such as the correct/incorrect assessment of an action or task at the individual or team level, and process data. Process data offer an insight into the interactional dynamics of the team members, which is important both for defining collaborative tasks and for evaluating the results of the collaboration (Morgan, Keshtkar, Duan, & Graesser, 2012; Morgan, Keshtkar, Graesser, & Shaffer, 2013).
The nature of the data from CPS depend upon the design, particularly if a human-agent or human-human approach is taken. In the human-human design, the data from CPS consist of time-stamped sequences of events registered in a log file. From a statistical perspective, these activity logs or log files are detailed time series describing the actions and interactions of the students. Interpreting actions and chats is quite complex, given the sheer volume and complexity of data generated in log files and dynamics of the students. If there are two people on a team, the actions of one of them will depend both on the actions of the other and on his or her own past actions. The statistical models used should accurately describe the dynamics of these interactions. These dynamics, which are defined by the interdependence between the individuals on the team, could also offer information that could be used to build a hypothesis about the strategy of the team.
In the human-agent design, many of these challenges are avoided by developing limited actions and interactions from which the student can select or perform. In essence, the multi-turn and multi-action interactions are converted to multiple-choice scoring events. The problems of interpreting and scoring open-ended human actions are circumvented by this multiple choice approach.
There are various analysis approaches that can be performed on the data from CPS assessments. These approaches include traditional psychometric scaling of items, problems, and problem clusters and factor analyses of the resultant matrix. Additional methods can be employed to analyze the wealth of data from CPS, including data mining tools to identify different patterns and strategies in problem solving, stochastic to identify different collaboration strategies, and, if the process data are more deterministic, aggregating the different paths to determine different partially correct scores.
These scoring and analysis possibilities will be discussed during the session.

Author

• Alina A. Von Davier, ACT, Inc.