Validation Evidence for Forced-Choice and Mixed-Format Knowledge Assessments

• In Event: Practical Guidance and Best Practices for Gathering Validity Evidence Based on Assessment Type

Abstract

The Assessing Students’ Progress on the Energy Concept (ASPECt) project is the context for the first paper in the session. The goal of the ASPECt project is to develop assessments to measure late elementary, middle, and high school students’ understanding of energy. It currently has two main components.
The first component is a set of three instruments that were constructed from a bank of 359 distractor‐driven, multiple-choice items to assess students’ progress on core energy ideas. The items are aligned to a learning progression for energy that is made up of three levels of complexity. The complexity levels progress from a basic level focusing on simple energy relationships and easily observable effects of energy processes to an intermediate level focusing on more complex energy concepts and applications to an advanced level focusing on still more complex energy concepts, often requiring an atomic/molecular model to explain phenomena (Herrmann-Abell & DeBoer, 2018). Validity evidence was collected through expert reviews, think aloud interviews and pilot tests with students in grades 4 through 12, and Rasch analysis of field test results from over 20,000 students. Results largely supported a model of students’ growth of understanding that progresses from an understanding of forms and transformations of energy to energy transfer to conservation while also progressing along a separate dimension of cognitive complexity.
The second component is a set of scenario-based tasks that are aligned to the three dimensions of science learning described in the Next Generation Science Standards (NGSS Lead States, 2013). These tasks require students to use science and engineering practices along with disciplinary core ideas and crosscutting concepts to make sense of energy-related phenomena. The tasks present a phenomenon or scenario followed by a series of constructed-response and multiple-choice items. The items within a task move students through a process of phenomenon/problem introduction, sense-making, and final resolution. Some items were aligned with one dimension, some with two, and some with three dimensions. When taken together, the items were intended to provide a comprehensive picture of students’ 3D understanding. To date, we have developed rubrics for tasks that are focused on assessing students’ ability to write explanations or arguments (Herrmann-Abell, Hardcastle, & DeBoer, 2020). The rubrics were based on the claim, evidence, reasoning (CER) framework (McNeill & Krajcik, 2011). Using Rasch analysis, we evaluated the rubric categories and found that rubric categories fit well to the Rasch model. Categories clustered in a hierarchy of difficulty in which reasoning and applying science idea categories were more difficult than citing evidence, which were more difficult than making a claim. The observed hierarchy in difficulty of CER categories is consistent with other studies and adds to the validity argument for the tasks as measures of students’ ability to write scientific explanations and arguments about energy-related phenomena. In addition, a panel of experts were consulted to evaluate the appropriateness of the task phenomenon/scenario, the alignment of the task to the three dimensions of NGSS, and the fairness and comprehensibility of the task.

Author

• Cari F. Herrmann-Abell, BSCS Science Learning