BackHome
Refresh: Off
Paper Summary
Share...

Direct link:

Performance Assessments to Validly Measure Cross-National University Economics Students’ Domain-Specific Critical Online Reasoning

Sat, April 13, 9:35 to 11:05am, Pennsylvania Convention Center, Floor: Level 100, Room 118C

Abstract

1. Objectives
Today, the use of online sources for informed decision-making in economics is becoming increasingly important. Prior research indicates that the demanded skillset can be described using the framework of domain-specific (DOM) critical online reasoning (COR). This paper presents this conceptual and assessment framework for developing computer-based performance tasks to (ecologically) validly measure DOM-COR skills in economics.
2. Theoretical Framework
Recently, the development of performance assessments to measure multifaceted constructs like 21st-century skills has steadily increased (Authors, 2019). Performance assessments are action-oriented tasks that use real-life professional requirements (e.g., creating lesson plans) as a basis for a realistic task scenario (Shavelson, et al., 2019) to simulate real-world decision-making in a holistic approach. The design of performance tasks to evaluate DOM-COR skills is based on the conceptual and assessment framework that incorporates three facets of DOM-COR, three levels of DOM-COR development, and three areas of requirements. This results in the 3x3x3 assessment taxonomy to:
• Measure the self-directed learning of university students, we developed a conceptual framework of COR-skillset, consisting of three facets: online information acquisition (OIA-skills), critical information evaluation (CIE-skills), and reasoning based on evidence, argumentation, and synthesis (REAS-skills) (Molerov et al., 2020).
• Measure the development of COR-skills in university studies, we follow the Model of Domain Learning by Alexander (2004), which differentiates between three levels: acclimation (basic level); competence (advanced level); proficiency (proficient level). In the assessment framework, we define behavioral anchors we expect at every level regarding every COR-facet.
• Model DOM-COR in economics, we differentiate between three requirement areas for professional activities:
o Select, evaluate, and verify the meaning of fundamental concepts and models like break-even point and supply-demand presented in online sources.
o Incorporate practical economic reasoning needed in professional decision-making (e.g., preparing a financial plan based on contradictory and/or uncertain business parameters.)
o Incorporate transdisciplinary reasoning to other disciplines and needed in decision-making about complex problems in naturalistic settings (e.g., cost-intensive lifesaving measures under limited resources).
These requirement areas build upon each other in their complexity, reflecting the development of DOM-COR among the three COR-facets and development levels.
3-4. Method and Data Source/Analysis
Using this framework, we develop four performance tasks that replicate real-life scenarios in economic contexts. These tasks require an online search for specialized information, its processing, and the utilization of that data to make reasoned decisions. The tasks are being pretested, piloted, and validated in the summer term 2023 and winter term 2023/2024 (pilot sample=100 students; main validation representative sample=600) at three large universities and six faculties across Germany and one faculty at a US-university (sample=200 students).
5. Results
We present initial results from the validation study regarding two validation criteria “internal test structure” and “relations to other variables” in a nomological network (according to Standards by AERA et al., 2014), and regarding international comparability and measurement invariance.
6. Scholarly Significance
The presented results will contribute to an important basis for establishing national and cross-national diagnostic and instructional approaches in higher education economics to foster DOM-COR skills.

Authors