Scoring and Dimensionality in Next Generation Science Standards Assessment

• In Event: 77.020 - Next Generation Science Standards Assessment Design Across Use Cases: From Formative Classroom Assessment to Large-Scale Accountability Assessment

Abstract

The Next Generation Science Standards (NGSS) are organized into three dimensions: Disciplinary Core Ideas, Science and Engineering Practices, and Crosscutting Concepts (NGSS Lead States, 2013). In line with NGSS instruction, it’s important to develop assessment that supports three-dimensional learning. However, some questions remain about how assessment can best accomplish this purpose. In particular, a single score may not be sufficient to capture the three-dimensional information in student performance (NRC, 2014). This research compares the impact of using unidimensional and multidimensional scoring and modeling approaches on multidimensional NGSS assessment items.

To examine the scoring and dimensionality of NGSS assessment items, 369 4th-6th grade students in the northeastern U.S. responded to items measuring 3 NGSS dimensions: Structure and Properties of Matter; Scale, Proportion, and Quantity; and Engaging in Argument from Evidence. Student responses were scored using a holistic rubric and a multidimensional rubric. Examples of each type of rubric are in Figures 1 and 2. Data from the holistic rubric was analyzed with a unidimensional Rasch model, and data from the multidimensional rubric was analyzed with a multidimensional Rasch model. All models were generated with ConQuest (Adams, Wu, & Wilson, 2015), version 4. Interrater reliability (IRR) was compared for each scoring rubric via the intraclass correlation coefficients (ICC; Shrout & Fleiss, 1979). To compare the psychometric models, model fit, reliability, standard errors and person fit statistics were examined, as well as the correlation between dimensions.

On the Structure and Properties of Matter and Scale, Proportion, and Quantity dimensions, multidimensional scores had higher IRR than holistic scores. On the Argumentation dimension, however, IRR was lower than that of the holistic scores (see Table 1). The correlations between the three dimensions were high, suggesting that separating them may not explain unique variance in student performance. However, student ability estimates varied depending on the dimension; for half of students, WLE estimates on different dimensions varied by a standard deviation or more. Reliability estimates from a unidimensional model were substantially higher than the multidimensional subscales (see Table 2). A multidimensional model allows for discrepancies in student performance across the dimensions, leading to better person fit for students with such discrepancies. Standard error of person estimates from the holistic and multidimensional models were similar. Even though the dimensional subscores focus on smaller aspects of the response, they provide a similar amount of information as the holistic scores, which ostensibly contain a broader reflection of student performance.

The results have different implications for different user groups. For teachers, tracking students’ progression is important for monitoring and guiding NGSS instruction. Multidimensional scoring rubrics serve this goal by providing information about student understanding on multiple dimensions of science learning. Measurement error was much higher when a multidimensional model was used. When researchers and evaluators are trying to estimate effects, unreliable assessment data adds noise. Low reliability also affects the precision of accountability measures like teacher value-added or student growth percentiles. However, concerns about measurement error should be weighed against the need for valid representation of multidimensional constructs.

Author

• Courtney Castle, The Woodrow Wilson Academy of Teaching and Learning