Contrasting Micro-Interaction Processes During Online Assessment of Collaborative Problem Solving: Case-Based Portraits of Two Dyads

• In Event: Developing Innovative Assessments of 21st-Century Skills: Online Inquiry and Collaboration

Abstract

This study recognizes the role and the quality of social aspects of collaborative problem solving (CPS). We ask how dyads build shared understanding and acquire CPS practices during online assessment in the Assessment and Teaching of 21st Century Skills (ATC21S; Griffin & Care, 2015) environment. As current technologies are inadequate to perform analysis of the content of online communication automatically, the aim of this study is to go beyond the individual CPS skills to interpersonal processes to explore interactions during CPS assessment (Authors, submitted). It is assumed that in spite of similarity in students’ CPS performance scores, variations in micro-interaction processes (Davis et al., 2015) may occur. Studying these patterns may provide insights into CPS and enhance understanding about how to better support acquisition of these skills.

In the ATC21S assessment environment, the individual student’s CPS competency in collaborative partnership is defined via a complex set of sub-skills, consisting of five broad skill classes of social and cognitive skills (Care et al., 2016; Hesse et al., 2015). Social skills (i.e. participation, perspective taking, social regulation) are about managing participants (including oneself). Cognitive skills (i.e. task regulation, knowledge building) are about managing the task. In high-quality CPS activity, social and cognitive skills are inherent, may overlap, and are implemented in accordance with situational needs.

The environment comprises a set of online interactive tasks in STEM domains. To fully understand the problem space and to identify necessary resources, students need to rely on their partner, with communication taking place via a free form chat interface (Care et al., 2015; Care et al., 2016). Scoring is based on participants’ actions, which include movement of artifacts and the occurrence of chat to collaborate. Students’ task completion generates logfile data and patterns in these data are automatically coded and scored according to the Rasch model, generating indicators of CPS elements (or subskills) and producing information on students’ social and cognitive skill levels (Adams et al., 2015; Harding & Griffin, 2016).

The research participants were first-year teacher education students (n=20) in a Finnish University, randomly assigned to their pairs. To better understand how CPS unfolds at the pair level, data includes CPS performance measures and logfiles from the assessment environment, complemented with retrospective reporting (CRR) data (van Gog et al., 2005), cued with screen recordings of the online assessment (Authors, 2016). Directed content analysis (e.g. Humble, 2009) was first applied on CRR data in which the categorization matrix applied the same behavioral indicators of CPS elements as used in the automated coding procedures (Adams et al., 2015). To determine micro-interactions at the pair level, logfiles were next compared to coded CRR data. For this paper, two contrasting cases, representing joint and individual problem solution orientations, are presented and visualized. We discuss how the identifications of problems in the appropriation and use of assessment practices online may inform design and refinement of practices.

Authors

• Johanna Pöysä-Tarhonen, University of Jyväskylä

• Esther Care, University of Melbourne

• Nafisa Awwal, University of Melbourne

• Päivi Häkkinen, University of Jyväskylä