BackHome
Refresh: Off
Paper Summary
Share...

Direct link:

A Performance and Response Process Data Triangulation to Identify Online Reasoning Patterns Among Graduates

Sat, April 13, 1:15 to 2:45pm, Pennsylvania Convention Center, Floor: Level 100, Room 116

Abstract

Objectives
The Internet is crucial when facilitating informal, post-university learning and providing up-to-date information for young professionals in various domains including medicine, law, and teaching[1]. Critical reasoning based on online information is vital for making well-informed decisions and taking appropriate data-based actions within these professions, especially considering the constant and rapid evolution of knowledge and practices[1,2,3]. Prior research highlights substantial deficiencies in under- and postgraduates’ critical online reasoning (COR) skills[4,5], however, there are few valid investigations of the online information-seeking abilities of young professionals when creating job-specific documents during professional training[21]. This study addresses this desideratum by profiling COR skills[6] that have been validly measured among young professionals from medicine, law, and teaching in periactional, computer-based, online performance assessments[7].
We aim to identify distinct profiles of COR during professional training by triangulating different data sources from COR assessments, including process data (e.g., URLs visited to solve COR tasks), rated performance data (i.e., written responses), and self-assessed general online media use and media trustworthiness. In this paper, we present findings from the first measurement at the beginning of professional training in medicine, law, and teaching.
Framework
COR is a skill set entailing the search, selection, critical evaluation, and processing of online information to solve everyday and job-specific problems (for more details, see proposal by [authors’ 3][4,6,21]).
Methods
COR was assessed remotely online within a virtual Microsoft Azure lab. In short, 20-minute COR tasks, participants were prompted to perform an open web search and use reliable information appropriately to write an open response while their log data and browser histories were recorded and further processed as an objective reference to participants’ COR performance ([5]; table 3). The process and performance data were evaluated by means of validated scoring rubrics from three trained raters (table 1). After the COR assessment, they were surveyed on their sociodemographic data and general online media use[4]. All these data sources underwent latent profile analyses (LPA) in Mplus[15] to classify the participants into homogeneous subgroups (tables 2-3).
Data
Two groups were identified in the LPA (N1=65; N2=79). Participants from Group 2 visited a lower number of unreliable websites on average. Similarly, Group 2 more frequently performed actions indicating a source and citation check. and consistently integrated more reliable websites into their responses (i.e., non-commercial, evidence-based, up-to-date, and renowned websites). This pattern also applies to the COR ratings for Group 2’s process data (incl. websites used), which reflect a higher source quality and source evaluation. Overall, Group 1 may be referred to as less online media-savvy than Group 2.
Results
Even though the groups are fairly homogenous in terms of their size, general media use, and general cognitive ability (final school and degree grade), the results indicate distinct information processing patterns between high vs. low COR performers. The performance and process data triangulation suggests that competent search and selection strategies are associated with the substantial differences in COR skills.
Significance
This study indicates that fostering online media awareness training among graduates could contribute to better COR skills.

Authors