Collaborative Problem Solving: Coding, Scoring, and Interpretation

• In Event: 31.089 - Knowledge Is Not Enough: Problem-Solving Competence and Innovative Skills for Modern Workplaces

Abstract

Objectives or purposes
The Assessment and Teaching of Twenty First Century Skills project (ACT21S) developed a process of constructing, scoring, calibrating indicators of social and cognitive competence among students completing a set of collaborative problem solving tasks. These calibrations were then used to estimate student ability parameters. The study also examines the stability of task parameters across countries that differ in culture and language.
Perspective(s) or theoretical framework
The study used a conceptual framework described by Hesse (2012) to define collaborative problem solving (CPS). This framework differed from that used by OECD (2012) for the 2015 PISA Program. Parameters were estimated in real time to enable reports of student ability which were then mapped onto a reporting module. IRT analysis was used to find the point of instructional intervention (Vygotsky, 1974) on a developmental continuum defining stages of increasing competence (Glaser, 1981).
Methods
(CPS) tasks were administered to dyads of students in sub-sets of three or four tasks with an estimated time demand of about 50 minutes. Data were collected from approximately 1000 students in Australian schools and compared to previous data from students in the six participating countries (Australia, Singapore, Netherlands, Finland, Costa Rica and the United States. The process of task development is described elsewhere (author).
The data analyses employed include differential item function analyses, examinations of distributions and independent analyses of the construct in each national data set. This paper will present the results of these analyses and discuss the implications of such analyses for other potential international studies planned for the future.
Data sources
As the students worked through the tasks, a log stream data file was generated. A search algorithm explored the log stream data to identify indicators of elements in a pre-defined collaborative problem solving conceptual framework developed by Hesse et al (2012). These indicators were coded and a new data file established containing 211 variables (indicators) of the Hesse collaborative problem solving elements. The scored data file was used to estimate indicator difficulty in one, two and five dimensions. This analysis reported describes the one-dimensional model.
The data from five countries was used to examine whether there is differential indicator behaviour across countries, and whether there are different interpretations of the underpinning construct.
Results
The data showed a high level of consistency across the participating countries. A sub set of indicators was identified that defined the same progression five countries.
Significance
Given that the task parameters are shown to be stable, it is feasible that cross country comparisons can be undertaken, but more importantly, within country teachers and other users of the assessment data can be confident that national or local data are not disturbed by differential task behaviour.
References
Glaser, R. (1981). The future of testing: A research agenda for cognitive psychology and psychometrics. American Psychologist, 36, 923-936.
Vygotsky, L. (1978). Mind in society (M. Cole, Trans.). Cambridge, MA: Harvard University Press.
Hesse, Friedrich, E. Care, J. Buder, K. Sassenberg, P. Griffin, (2012). A Framework for Teachable Collaborative Problem Solving Skills. http://atc21s.org/index.php/resources/white-papers/#item7

Author

• Patrick E. Griffin, University of Melbourne