Leveraging Affordances of Technology-Enhanced Items for Assessment of Three-Dimensional Science Proficiency

• In Event: 77.020 - Next Generation Science Standards Assessment Design Across Use Cases: From Formative Classroom Assessment to Large-Scale Accountability Assessment

Abstract

The Next Generation Science Standards (NGSS; NGSS Lead States, 2013) redefines science proficiency as an integration of multiple dimensions: science and engineering practices (SEPs), disciplinary core ideas (DCIs), and cross-cutting concepts (CCCs). Large-scale assessments that can measure both the depth and breadth of such three-dimensional science proficiency (3DSP) are necessary to monitor the impact of NGSS-aligned reform (NRC, 2014). Discrete items are essential for measurement of the breadth of the NGSS; however, there is a lack of foundational theory for how discrete items can be used to assess 3DSP. In this presentation, we will discuss the development of one approach to discrete assessment of 3DSP.

Measurement of 3DSP requires students to demonstrate their understanding of a DCI through engagement with an SEP (Achieve, 2018). SEP engagement is active and generative by nature: students must for example, develop a model or engage in argument (NRC, 2012). Valid discrete assessment of 3DSP therefore requires item types that allow students to construct their response. Open construction presents challenges both for efficiency of administration and scoring at scale. Technology-enhanced (TE) items provide an intermediate level of constraint between selection and open construction (Scalise & Giffords, 2006), and thus a balance between face validity and efficiency of items. This makes them good candidates for discrete assessment of 3DSP.

We used an iterative evidence-centered design process (Mislevy & Haertel, 2006) to define a task model for discrete assessment of performance expectations (PEs) associated with SEP2 (developing and using models) and SEP7 (engaging in argument from evidence). We conducted a review of literature to inform definition of a rich but narrow slice of each practice. We then evaluated these construct definitions against the affordances of TE item types. This resulted in identification of the match (drag-and-drop) item type as well-aligned to the cognitive demands of both practices. We defined general characteristics of match items used for assessment of modeling and argumentation, including two sets of scoring rules. One set of scoring rules evaluates the 3D performance overall, while the second identifies the separate contributions of the DCI and SEP. We used the resulting task models to guide development of 24 items that assess the intersection of the modeling and argumentation practices with the life sciences and physical sciences domains.

Cognitive labs conducted with 20 middle school students with four items developed from the task models indicate that the item approach is promising for assessment of 3DSP. Data showed that student performance on the items was affected both by their understanding of the DCI and by their proficiency with the SEP. An upcoming administration of the full set of items to 1500 students will examine the reliability of this measure and the generalizability of practice proficiency across domains. These results will be discussed during the symposium.

Discrete items are needed to sample the full breadth of the NGSS. Evaluation of cognition associated with an SEP against affordances of technology-enhanced item types, resulted in definition of an approach to discrete item development that shows promise for measurement of 3DSP.

Authors

• Janet Koster van Groos, Educational Testing Service

• Katherine Furgol Castellano, Educational Testing Service