Evaluating Item Quality in Next Generation Science Standards Assessments

• In Event: 16.023 - Developing Design Principles for Improving State Assessment Items to Support More Valid and Equitable Measurement

Abstract

Purpose & Significance
In recent years there have been fundamental changes in how K-12 science education is conceptualized. Specifically, expectations of what it means to be competent in doing science and understanding science have broadened. Many states are now translating this more contemporary view of science education into standards such as the Next Generation Science Standards or modified versions of the NGSS. In the NGSS, learning outcomes are expressed as expectations that integrate both the competencies (i.e., skills and practices) and content understandings. However, decades of development of assessments designed to evaluate content knowledge separately from science skills or practices have left teachers, education administrators, and assessment developers with few resources suitable for use in this new paradigm (Hannaway & Hamilton, 2008; NRC, 2011, 2014; Pellegrino, 2013).
The review points to the specific characteristics of each assessment and task that align with the three dimensions of science and which aspects or dimensions need more attention. The outcomes of this work show how new assessments will need to differ from previous assessments and how existing assessments could be used as models for development of new assessments that are more fully aligned to the NGSS and similar frameworks.
Theoretical Framework
Our assessment item review criteria were largely based on the goals described in the following sources: the K-12 Framework for Science Education (NRC, 2012), the NGSS (Achieve, 2013), Developing Assessments for the Next Generation Science Standards (NRC, 2014), and the Guide to Implementing the Next Generation Science Standards (NRC, 2015). Additional criteria that derive from general principles for high quality assessment came from sources such as Knowing What Students Know (NRC, 2001b).

Methods & Sources
To identify possible model science assessment items and tasks, our assessment team conducted a three-stage review and analysis. 1) We established a bank of existing assessment items; 2) We developed review criteria, and 3) We conducted an in-depth analysis of a subset of items and tasks. We established a bank of existing assessments from which we could cull potential model items and tasks. The assessments we chose exhibited the following characteristics: they reflected the range of assessment formats that would be part of the new system of assessments (i.e., selected-response items, short constructed-response items, and short and extended performance tasks); they were designed for a variety of purposes (e.g., curriculum-embedded, external summative); and they were aligned to constructs relevant to the newer conceptions of science learning (i.e., disciplinary core ideas, science and engineering practices, and crosscutting concepts).
Findings
We have identified four key features of assessments needed to support the newer vision of science learning and have provided examples of items that illustrate promising approaches to meeting the goals of the NGSS and similar state science frameworks:
• Assessments should be aligned with and integrate multiple learning dimensions of science and engineering
• Assessments should focus on the big ideas in science
• Assessments should address the full range of science and engineering practices
• Assessments should require students to demonstrate their reasoning and problem-solving skills.

Authors

• Jill Wertheim, Stanford University

• Nicole Holthuis, Stanford University

• Susan E. Schultz, Stanford University