Developing a Collaborative Problem-Solving Assessment for NAEP

• In Event: 53.033 - Investigating the Inclusion of Collaborative Problem Solving in the Nation's Report Card

Abstract

The item development for any Collaborative Problem Solving (CPS) assessment is very challenging for different reasons – implementing the complexity of the framework, determining the item characteristics, or addressing the scoring issues, are some of the necessary, but difficult steps.
While the development process can be based on known procedures for test item development, the major difference is the definition of a “test item”. In CPS, the “items” are intricate. It is feasible that each “item” had multiple solutions and more than one student may be involved in finding the solution. Given the assessment is concerned with the process of collaboration as much as it is concerned with the solution, consideration of the process must be taken into account in the development. To this point, the tasks should involve elements of ambiguity.
In addition, the tasks must be designed and developed to facilitate and encourage collaboration (either between students or with the agent). For example, students should have use and control of multiple resources and the task should engage students in ways that ensured they were dependent on one another (or dependent on the agent) for successful resolution. For human-human settings, no one student should be able to complete a task alone and a task should not be solvable unless every student contributed to the problem solution. For this reason, each student should have unique control of a resource that was essential to the problem solution. That is, each of the students should access or control different but essential data and resources.
While the specific development steps may vary, iterative prototyping and development stages are required in the development process. After identification of possible problems for inclusion in the assessment, the next step may be human-human interaction on the potential problems using computer-mediated communication, face-to-face interactions, or think aloud procedures. This step allows developers to capture the interactions and verbal protocols during problem solving and uncover strategies, collaboration approaches, and distributions of actions. Developers can then use this information to help structure the task, associated test items, and scoring rubrics.
These development considerations, in light of the design decisions, will be discussed during the session.

Author

• Arthur C. Graesser, The University of Memphis