Comparing Arithmetic and Holistic Coding of Performance Assessment Essays

• In Event: Conceptual and Methodological Challenges in Comparability of International Performance Assessment of Critical Thinking

Abstract

1. Purpose
This study asked whether holistic and arithmetic coding are consistent (i.e., produce comparable overall scores despite being measured in different metrics and with limited sub-score congruence). Arithmetic coding requires an exhaustive list of arguments across all documents and is therefore limited to PA tasks with a given set of documents (usually 7-8). In contrast, holistic coding can be used for limited and unlimited (e.g., free document search) PA tasks. It is therefore possible to code essay responses to PA tasks with both methods.
2. Theoretical Background
Performance Assessment (PA) Tasks have emerged as a valid measure of the development of critical thinking in college on an aggregated level (i.e., for cohort and institutional comparison; Braun, 2019). Efforts to create valid measures across nations is more challenging as cultural differences affect the familiarity with the subject of the PA task, its contextual relevance, and translation effects (Ronderos et al., 2021).
Another relevant aspect of international comparisons is the consistency of the coding. The basis of the PA assessments are written essays that are coded either through a coding rubric (“holistic coding,” see Zlatkin-Troitschanskaia et al., 2019) or a direct coding procedure that analyzes clauses with respect to arguments and source assessment (“arithmetic count-based coding,” see Authors anonymous, in prep). The advantage of arithmetic count-based coding for international PA comparisons lies in the fact that it is potentially less affected by semantics and language interpretation compared to the holistic coding approach. It also increases the transparency of the coding since the relevant codes are marked in the documents themselves, allowing researchers to review the exact clauses that earned each code and to make any necessary revisions.
3.-4. Data and Methods
Five essays that were coded arithmetically at a university in the United States as part of a larger research project were subsequently coded holistically by an expert coder at the University of Mainz in Germany using the original English essays.
5. Results
The correlation of the composite arithmetic score and the average holistic score was r = .94. Given the interrater reliability for both coding methods, the true correlation is close to 1. The findings underscore that arithmetic coding is equivalent to holistic coding, at least at the level of total scores. The small sample does not yet allow a discussion of differences or convergence at the level of sub-scores.
6. Scientific Significance
The finding suggests that arithmetic coding has strong potential for international comparisons of critical thinking as the coding has shown high interrater reliability even with short training.

Authors

• Blake Ebright, University of Michigan

• Dominik Braunheim, Johannes Gutenberg University of Mainz

• Kai S. Cortina, University of Michigan