Measuring Students' Exposure to Standards-Aligned STEM Content and Practices

• In Event: Developing National Indicators for K–12 STEM Curriculum and Instruction

Abstract

Purpose and Framework

Efforts to address Indicator 5 (classroom coverage of content and practices in CCSS-M and NGSS) face several hurdles. First, existing methods such as surveys, observations, and artifacts have logistical and technical challenges (Goe, Bell, & Little, 2008; Pianta et al., 2009). Second, measures validated for STEM disciplines other than mathematics are limited. Finally, measuring within-classroom differences in students’ experiences is particularly challenging in technology-based, personalized learning (PL) contexts (BMFG, 2014), which are increasingly prevalent. This paper explores new measures that could be adopted on a large scale to collect evidence about instructional content and practices aligned with CCSS-M and NGSS, with an emphasis on innovative approaches to addressing within-classroom differences. It addresses three questions:

1. What methods have been used to measure exposure to content and practices in STEM, and to what extent have these measures demonstrated good technical properties and alignment with CCSS-M and NGSS?
2. What innovative data collection methods used in other areas could be adapted to STEM contexts?
3. Which methods are suitable for PL environments?

As with other papers in this session, this paper is grounded in the idea that documenting students’ opportunity to learn (OTL) is crucial for assessing STEM education in the U.S. There are two potential perspectives to take when considering OTL. The first focuses on what teachers do in the classroom, whereas the second emphasizes what students do; i.e., their learning experiences. These perspectives overlap but are not identical, particularly in PL contexts. Although measures developed under both perspectives provide useful information about instructional exposure, measures that focus on students’ experience with desired content and practices might be more closely related to students’ learning outcomes and therefore more important for understanding the quality of STEM education and how to improve it (Haystead, 2010; Priest et al., 2012).

Methods

This research draws on interviews with experts on STEM instruction, assessment, and technology and on a review of existing measures and innovative data-collection approaches. The latter included approaches such as event sampling methodology, which has been used in several public health projects and which has promise as a tool for gathering student-level information about instruction.

Results

We identified several innovative data-collection methods that will be explored in detail in this presentation. These include new formats for student and staff surveys about teaching and learning, technology-based curriculum resources with data-capture features, and electronic games for STEM learning and assessment that document how students spent their time and the complexity of problems and content to which they were exposed. Most of these would be challenging to implement on a large scale in their current form but could be feasible under certain time and sampling schemes.

Significance

Understanding whether high-quality STEM educational opportunities are being provided to all students requires classroom-level evidence of the content and quality of instructional opportunities, regardless of whether those opportunities are provided through traditional instruction, intelligent tutoring software, or some other means. This research will provide guidance for policymakers and others who wish to gather such evidence.

Authors

• Laura S. Hamilton, RAND Corporation

• Kun Yuan, RAND Corporation

• Brian Stecher, The RAND Corporation