Operationalizing the Science and Engineering Practices (SCIOPS)

• In Event: Developing National Indicators for K–12 STEM Curriculum and Instruction

Abstract

Purpose

NRC’s Monitoring Progress report (National Research Council, 2013) highlights a number of areas in which development work is needed to realize the goal of an indicator system for K–12 STEM education. Especially challenging is the need to develop methods to measure the extent to which classroom instruction aligns with the Framework for K–12 Science Education (National Research Council, 2012) and the subsequently released Next Generation Science Standards (Next Generation Science Standards [NGSS] Lead States, 2013), particularly the science and engineering practices.

This paper will describe the development of two products to facilitate measurement of Indicator 5. The first product is a primer that operationalizes the science and engineering practices for different grade levels (i.e., early elementary, upper elementary, middle grades, and high school) and content areas. The second product is a selected-response survey for measuring teachers’ use of the practices.

Framework

One way of assessing students’ opportunity to learn is by examining what teachers are doing in their classrooms. However, there is little existing guidance for teachers on how the science and engineering practices in the NGSS should be taught to students. Further, there are no tools for measuring the extent to which classroom instruction provides students’ opportunities to engage with the science and engineering practices described in the NGSS on a large scale cost effectively. This project addresses both of these challenges.

Methods/Data

To date, the project has facilitated a four-round modified-Delphi panel focused on: (1) constructing definitions of each science practice; (2) identifying the key elements of what students need to experience at various grade levels that are essential for helping them master each science practice; and (3) generating examples of what instruction “looks like” as students engage with each science practice.

Panel responses are currently being analyzed and will be used to inform the development of a primer for operationalizing the science practices. The primer will then be used to guide the creation of survey items assessing how, and how often, teachers implement the science practices described in the NGSS in their instruction. The project will follow a similar process for the engineering practices.

Results

The paper will focus on what has been learned during the development of the primer. For example, preliminary analysis of panel data indicates that the key elements of instruction students must experience to master each science practice are similar across areas of science, but differ across grade bands.

The paper will also describe the development of survey items including sources of evidence for reliability and validity (e.g., cognitive interviews, expert review, large-scale piloting). In addition, it will present evidence of the dimensionality of the questionnaire (e.g., from factor analyses).

Significance

This project aims to develop a consensus among STEM educators about operational definitions of the science and engineering practices at different grade levels. These definitions will move the field forward substantially. In addition, the project will use the consensus definitions to draft, pilot, and validate survey items that can be used cost effectively at scale to measure Indicator 5.

Authors

• Peggy Trygstad, Horizon Research, Inc.

• Eric Banilower, Horizon Research, Inc.

• Joan D. Pasley, Horizon Research, Inc.