University Students' Dealing With Uncertainty and Ambiguity While Assessing Multiple Contradictory Sources: Research and Practice

• In Event: Addressing Practical Challenges in the International Performance Assessment of Critical Thinking

Abstract

1. Objectives/Purposes
Critically reasoning based on contradictory information from multiple sources (CT) is a crucial ability in a digital age (Shavelson et al., 2019). iPAL follows an innovative approach, measuring CT of higher education students by using performance tasks that simulate real-life situations requiring decision-making and judgment students may encounter in academic and professional domains. Based on the iPAL framework, we developed a performance task (PT) with a case scenario (renewable energy) comprising 22 (ir-)relevant, (un-)reliable, and partly conflicting pieces of information (Zlatkin-Troitschanskaia et al., 2019). After solving the PT, we conducted semi-structured cognitive interviews with 30 advanced higher education students selected based on specific criteria to understand how they integrated and evaluated this information in their decision-making and which factors influenced their reasoning. In particular, we focused on instances where students struggled to assess a source or addressed a source that opposed their own opinions and beliefs.
2. Theoretical Framework
The PT assesses CT in terms of five theoretically driven facets according to the construct definition: participants’ identification, evaluation, and integration of data sources; recognizing and using arguments; developing one’s own argument; recognizing consequences of decision-making; and taking appropriate communicative action (Zlatkin-Troitschanskaia et al., 2019). Validity evidence is gathered by evaluating the students’ task responses (short essays), cognitive interviews, and an additional questionnaire pertaining to students’ individual characteristics.
3. Method
The PT requires students to consider and critically analyze 22 sources with varying degrees of trustworthiness and relevance. The qualitative analysis of the cognitive interviews focuses on CT in moments of uncertainty regarding the content of a source.
4. Data Source/Analysis
In addition to cognitive interview data regarding students’ ability to deal with irrelevant and unreliable information, test performance was scored using a newly developed 6-point Likert-type anchored rating scheme based on a CT construct regarding the 5 facets and 23 subdimensions (Zlatkin-Troitschanskaia et al., 2019). Students were also asked to rate their own task performance.
5. Results
The students reported when they made their decision on the task. A significant number (~25%) made their decision before processing the information. Sources were assessed selectively and many did not factor into students’ reasoning and decision-making. Students reported why they neglected/excluded some sources and which sources were confusing to them. Two primary reasons were a lack of understanding and/or a lack of belief in content contradicting one’s own opinion. Data analysis profiled specific strategies during task-solving. A major differential factor between these profiles was that most students completely discarded all sources that confused or challenged them.
6. Scholarly Significance
Our findings indicate that moments of uncertainty reflect exposure to often complex, but ultimately new and productive information. The identified reasons for students’ uncertainty provide important insights about the strategies of how they manage uncertainty during source evaluation. Our study points to yet-to-be-researched aspects of information comprehension in today’s learning environments and derives hypotheses on specific strategy profiles for future research. Regarding practical implications, we illustrate opportunities and challenges for using PT in teaching interventions to foster students’ CT in higher education.

Authors

• Dominik Braunheim, Johannes Gutenberg University

• Olga Zlatkin-Troitschanskaia, Johannes Gutenberg University of Mainz

• Klaus Beck, Johannes Gutenberg University