Measurement of Collaborative Problem-Solving Skills

• In Event: Roundtable Session 60
  In Roundtable Session: Assessment of Technology-Based Collaborative Skills: Transforming Assessment for the 21st Century

Abstract

Objectives

This presentation will outline the process of defining, operationalising and measuring collaborative problem-solving. It explores the implications of the process of collaboration on problem-solving. It also explores methodological issues of assessment and measurement of individuals working in collaborative group processes.

Theoretical framework

Collaborative problem-solving has been conceptualised as consisting of five broad strands: participation as a member of a group by contributing knowledge; using experience and expertise in a constructive way; recognising the perspective of others in a group; recognising the need to manage contributions within the group; identifying and regulating the problem space; and building knowledge and understanding.

Methods and sources

In the process of developing collaborative problem-solving scenarios, three types of scenarios and tasks were trialled. The first type was short term problems related to school curriculum subjects mathematics, science and critical thinking. A second type was short dyadic resolution tasks based on experimentation, but not directly related to school curriculum. In these problems, there was a clear problem definition and process to reach the solution. A third classification adjusted a serious game with no direct connection to school curriculum, no clear problem definition, ambiguous instructions, and a requirement and dependence on collaboration, problem definition and exploration. Each task was checked with teachers to ensure they were realistic, that students would be able to work with them, that the tasks could differentiate between high and low levels of ability, and that the skills underpinning the task resolution were teachable. ‘Think-aloud’ protocols were used in small-scale studies (cognitive laboratories) with students representing the target population in order to generate ideas for automatic coding and scoring of student performances on the tasks.

Data sources

A series of small-scale pilot studies was undertaken using intact classes to determine the technology and administrative requirements for implementation and assessment administration. They represented a rehearsal for the large-scale trials that have been carried out in seven countries. Trials were conducted in seven countries across five languages and data were collected using a matrix sampling approach to maximise calibration accuracy.

Results and substantive conclusions

This study did not compare countries. Its remit was to develop and calibrate tasks and investigate the influence of language, curriculum and computer experience on task performance. Calibration of the tasks yielded a series of developmental continua which were then compared to the original hypothesised construct to identify any substantive shifts in the debate on the nature of the construct being measured. This presentation provides the qualitative results from both cognitive laboratories and quantitative results from field trials, and points the way to future enhancement of measures such as these.

Scientific and scholarly significance

This was the first study that explored methodological issues involved in the measurement of collaborative problem-solving. It has shown that it is possible to both construct and calibrate such tasks and to inform teaching sequences that help to improve individual student performances in both collaboration and problem-solving.

Authors

• Patrick E. Griffin, University of Melbourne

• Esther Care, University of Melbourne