Variability Among Students From Different Majors in Critically Dealing With Multiple Information Sources

• In Event: Conceptual and Methodological Challenges of International Performance Assessment of Critical Thinking

Abstract

Objectives
The ability to critically assess and reason with multiple (partly contradictory) sources (CR) is increasingly important for success in higher education (Shavelson et al., 2019). This skillset seems to be fostered in study programs to a widely varying (and often limited) extent. (Zlatkin-Troitschanskaia et al., 2019a). To validly measure CR among university students, the iPAL project features a performance task (PT) simulating real-life decision-making situations, comprising 22 (ir-)relevant, (un-)reliable and partially conflicting sources (Zlatkin-Troitschanskaia et al., 2019b), that students are asked to deal with. Semi-structured cognitive interviews with 30 students from different majors at different stages of higher education were conducted after they completed the PT. The present analysis focuses on performance variation among students of different majors at varying stages of their studies.
Theoretical framework
The PT design is based on the construct definition of CR, which consists of five theory-derived facets: participants’ identification, evaluation, and integration of information sources; recognizing and using arguments provided in the sources; developing own arguments; recognizing consequences of decision-making; taking appropriate communicative action (Zlatkin-Troitschanskaia et al., 2019b). Validity evidence is gathered from the students’ task responses (short essays), cognitive interviews, and a questionnaire on students’ individual characteristics.
Method
The PT requires students to consider and critically analyse 22 sources with varying degrees of credibility and relevance. Quantitative analysis is focussed on differences in student performance depending on study semester and domain. Qualitative analysis of students’ text responses and the cognitive interviews provides insight into the task-solving process including information-processing and argumentation in decision-making, which may change over the course of studies and differ between majors.
Data Sources and Analysis
The sample features students in their 2nd to 11th semesters: mean value=5.13 study semesters, standard deviation=2.25. Most of the students were enrolled in the study program of economics-education (N=18), while the remaining 12 were enrolled in different programmes (e.g., psychology, geography).
Students’ PT performance was scored using a 6-point Likert-type anchored rating scheme based on a CR construct definition entailing 5 facets and 23 subdimensions (Zlatkin-Troitschanskaia et al., 2019b). The cognitive interview protocols were coded and evaluated using a newly developed coding scheme and qualitative content analysis.
Results
The full sample scored an average of 3.52 out of a maximum of 6 points (SD=0.73), suggesting sufficient variance in PT performance. Correlative analysis between study length and PT performance showed that master students outperformed bachelor students. Overall, students of economics education performed worse than students from other programs (e.g., psychology); however, these correlations are not significant due to small sample sizes and are best interpreted by qualitative analyses of the cognitive interview materials with regard to possible reasons for these differences.
Significance
Our analyses provide initial insights into the differences in level and development of CR skills among students in different study programs. The results also suggest that PTs are sensitive CR? differences. The findings are in line with prior research, which indicates an overall insufficient and limited fostering of CR skills in university studies.

Authors

• Olga Zlatkin-Troitschanskaia, Johannes Gutenberg University of Mainz
  Non-Presenting Author

• Dominik Braunheim, Johannes Gutenberg University of Mainz
  Presenting Author

• Klaus Beck, Johannes Gutenberg University
  Non-Presenting Author