Digital Performance Assessment for Teaching and Measuring Critical Thinking in the Domain of Data Literacy

• In Event: Addressing Practical Challenges in the International Performance Assessment of Critical Thinking

Abstract

1. Objectives
Following the growth of performance assessment design (Shavelson et al., 2019), the purpose of the present study was to design and validate a digital performance assessment for teaching and measuring critical thinking (CT) skills in the domain of data literacy. Data literacy skills are considered essential for participating in the 21st century as new technologies have led to an exponential increase in the volume of data available in daily life (Bonikowska et al., 2019). CT is essential for data-literate individuals to identify issues and challenges associated with data (Leighton et al.,2021). Digital performance assessments allow students to interact with a simulated digital environment to perform complex tasks to demonstrate skills that resemble real-life situations (AERA et al., 2014).
2. Theoretical Framework: Design
Guided by next generation performance assessments (Shavelson et al., 2019) and Evidence-Centered Design (ECD; Mislevy, 2006; Mislevy et al., 2003), the present digital performance assessment was designed to elicit evidence to support inferences about students’ acquisition of data literacy. Risdale et al. (2015) describe data literacy skills as collecting, discriminating, managing, evaluating, and applying information in a critical manner to make effective decisions in knowledge-based and technologically rich economies. Tasks were developed to follow the overall four-step structure described by Braun et al. (2020) to measure three CT aspects of data literacy: 1) analyzing and evaluating the relevance and credibility of information from different sources; 2) interpreting information and extracting relevant claims and supported evidence; and 3) synthesizing claims, evidence and making informed judgments or decisions.
3. Methods
Preliminary validation of this newly developed PA involves a purposeful sample of 50-60 undergraduate students from a Canadian university. Students participate in think-aloud interviews as they engage with PA tasks to test their critical thinking in this domain. The sample size is appropriate for identifying stable categories of response processes (Leighton, 2017; Lutsyk & Leighton, 2018). Students also complete online questionnaires that test their learning strategies and task comprehension.
4. Analysis, Data Sources and Results
A coding manual based on the three facets of critical thinking is used to categorize students’ verbal reports. Inter-rater agreement (i.e., Cohen’s kappa) of at least 0.80 is sought. Students’ verbal reports are the primary data sources, supplemented by responses to online questionnaires. A key aspect of the results focuses on linking the PA blueprint, design, and multiple data sources to create an evidentiary argument of the PA’s efficacy in eliciting the construct of CT, and how tasks can be improved.
5. Scholarly Significance
The present study advances our knowledge about methods to design digital performance assessments of the CT processes underlying data literacy. The paper and presentation therefore focus on the (1) digital assessment blueprint, (2) design of data literacy tasks, and (3) initial validation of response processes elicited with the tasks designed. Within each of these foci, challenges, lessons, and future directions are discussed to help others avoid pitfalls.

Authors

• Jacqueline P. Leighton, University of Alberta

• Maria Cutumisu, University of Alberta

• Fu Chen, University of Macau