Measuring Critical Thinking Through Performance Assessment Tasks: A New Approach to Essay Coding

• In Event: Conceptual and Methodological Challenges of International Performance Assessment of Critical Thinking

Abstract

Objective or Purposes
A promising approach towards a nuanced measurement of critical thinking (CT) subcomponents was proposed by Zlatkin-Troitschanskaia et al. (2019) who demonstrated sufficient interrater reliabilities for the coding of student essays in CT performance tasks (PT). However, the validity of the assessment remains contingent on substantial coder training. Our alternative coding approach (“Arithmetic Coding”, AC) is based on counts and does not involve rating scales.
Theoretical Framework
Definitions of CT in cognitive psychology tend to be narrowly based on correct information processing which lends itself to experimental research designs (e.g., System 2 reasoning, Kahneman, 2011). Educational psychologists tend to prefer a wider definition that includes the skill to communicate the deliberative process in authentic real-word situations (Shavelson, et al., 2019). College education increases subject-specific critical thinking, e.g., epistemological reasoning in advanced natural science classes. The moderate effect size of around .5 SD in CT gains for four-year college education also extends to tasks designed to measure CT in the broader sense (Mayhew, Rockenbach, Bowman et al., 2016, Pascarella & Terenzini, 1991; 2005).
The general perspective about the impact of college education on CT disregards the multifaceted nature of the broader construct. Referring to the iPAL (International Performance Assessment of Learning collaborative) assessment framework (Braun et al.,2020) highlights several subskills that are unique to the domain-unspecific conceptualization of the construct: Trustworthiness of an information source is rarely an important aspect in scientific reasoning, but is a key aspect of everyday CT, as is reflection of potential biases of sources. Those aspects can be assessed using curated tasks (Performance Tasks, PT) based on real-life problems of societal relevance (e.g., immigration, alternative energy etc.).
Method
Our starting point was the identification of all arguments that are explicitly stated or can be derived from the documents provided. To keep this feasible, we developed two PTs with fewer documents with ca. 30 arguments, half in favor, half opposed to a controversial decision. Subjects had 60 minutes to write a short essay to elaborate on their position on the issue and how they would proceed.
4. Data Source and Analysis
In two one-semester studies (N1pre/post = 99, N2 cross-sectional = 57), we identified for each student all the arguments mentioned, counted reflection on trustworthiness and relevance of the source, and assessed the writing skills (spelling, grammar, text structure).
5. Results
Our multidimensional analysis demonstrated that a) students tended to favor arguments that were grounded in strong moral imperatives, a trend that did not change by the end of the semester; b) Critical thinking developed only along one axis of our operationalization: the ability to see more arguments with a better balance of pro and con arguments. Students did not improve source analysis skills or writing skills over one semester.
6. Scholarly Significance
This work demonstrates the potential to make the coding of PTs more transparent and efficient without loss of precision and nuance.

Authors

• Kai S. Cortina, University of Michigan
  Presenting Author

• Blake Ebright, University of Michigan
  Non-Presenting Author