Constructing Performance Assessments for Formative and Summative Use in Undergraduate Colombian History

• In Event: Conceptual and Methodological Challenges of International Performance Assessment of Critical Thinking

Abstract

Objectives
Developing critical thinking (CT) performance tasks (PT) to be used as formative assessment and to obtain summative measures is an aim of IPAL (Braun et al., 2020). This project aims to develop such a CT PT and incorporate it into a course on Colombian history taken by all undergraduate students and taught by faculty from all schools at Los Andes University.
Formatively, the PT provides teachers and students a tool for developing CT which introduces them to the different facets of critical thinking. For summative purposes, the PT produces valid and reliable performance measures of CT to be used as base line to gauge the different undergraduate programs´ value added.
Theoretical Framework: Design
iPAL’s critical thinking framework (Braun et al., 2020) and Bermudez’s (2015) as proposed in Mejía et al. (2019) define the construct. For assessment development we use Evidence Centered Design (ECD) (Mislevy & Haertel, 2006).
CT is a multifaceted construct that comprises conceptualizing, analyzing, drawing inferences or synthesizing information, evaluating claims, and applying the results to various purposes (Shavelson et al., 2019). This construct has three main facets: Analyze and evaluate information; recognize and consider consequences; communicate a coherent argument with a position/course of action. Mejía et al (2019) elaborated on the iPAL framework by incorporating a perspective analysis facet.
Methods
The PT’s was selected to be a complex, actual issue in Colombian politics: the definition of the border between agricultural and preservation lands. The PT involves economic, social, and environmental issues. Documents were taken from real sources and adapted by selecting extracts. The resulting pieces, together with information about the authors and the context, and with three alternative sets of instructions were trialed in cognitive laboratories with students. Focus groups were used to understand students´ difficulties with the instructions and document library. After scoring a set of answers to the PT we adjusted the general rubric and instructions.
A pilot application will take place next semester using a platform that allows for online scoring and scoring quality control.
A general rubric was developed to be adapted for scoring different types of tasks. Validation was carried out through expert reviews and pilot applications.
We developed materials to introduce critical thinking to professors and students.
Analysis, Data Sources
Analysis and decision making for the development of the PT is based on results from the focus groups, cognitive laboratories, discussions with experts and pilot test.
Results
At this point we have found that:
• Understanding the complexity of a problem is a central aspect of CT, and requiring students to state the problem and its complexities before proposing a solution, makes their answers richer and elicits CT.
• Students get more engaged in the task if the document library includes documents in different formats (Articles, news, tweets, videos, maps)
Scholarly Significance
The present study helps advance our knowledge about methods to foster and measure the development of complex competencies such as CT in undergraduate programs.

Authors

• Julian Patricio Mariño von Hildebrand, Universidad de los Andes
  Presenting Author

• Adriana Molina
  Non-Presenting Author