Targeted Linguistic Modifications of Science Items for English Learners

• In Event: Validity Issues in Assessing English Learners

Abstract

Objectives. The goal of this study is to investigate the effect of linguistic and visual modifications of Grade 5 multiple-choice science test items on ELs and non-ELs’ performance.

Perspectives. The science tests administered by schools in the U.S. to fulfill the requirements of the No Child Left Behind Act have high-stakes consequences for students, teachers, schools, and districts. Most states assess ELs’ science knowledge and skills with tests written in English in students’ first year in U.S. schools, even though ELs are still developing their English proficiency (Solórzano, 2008; Wolf et al., 2008). Many researchers have questioned whether such tests are valid measures of ELs’ knowledge and skills in science (Sireci & Faulkner-Bond, 2015).

Researchers investigating the interactions of ELs with the language of test items have demonstrated that ELs’ test scores are affected by specific features of the language of test items, independent of item content. The majority of this research has focused on mathematics test items, and the current study extends this work to science test items.

Method. Targeted linguistic modifications of test items were used to identify the linguistic and visual features (e.g., unfamiliar vocabulary) of state science test items that have the largest effects on the test scores of ELs. Modified versions of 20 released Grade 5 multiple-choice state science test items were created. Items were modified in four subsets of five items each, with all items in a subset undergoing the same type of modification, while the science content was kept constant. These targeted modifications tested four distinct hypotheses about the effects of specific linguistic and visual features of test items on ELs’ performance.

Over 2,200 Grade 5 ELs and non-ELs were given tests consisting of original and modified test items. Raw score, IRT score, test characteristic curve, and DIF analyses were used to investigate differences between EL and non-EL performance on original and modified test items at the level of the test, the test item subset, and the individual test item.

Results. Two types of item modifications were most successful in improving performance for ELs—adding visuals (especially in the answer choices) and modifications based on feedback from students in cognitive labs. Within these modification types, three individual item modifications with common features emerged as the most successful. Details of these modifications will be provided in the presentation.

Significance. Tests following in the wake of the Next Generation Science Standards are expected to have increased language demands (National Research Council, 2014), potentially leading to increasing test score gaps between ELs and non-ELs as English language proficiency is increasingly measured by such tests. For this reason, it is essential for test item design to be informed by research on the features of test items that have the largest impacts on the performance of ELs.

Authors

• Tracy E. Noble, TERC

• Stephen G. Sireci, University of Massachusetts - Amherst

• Craig S. Wells, University of Massachusetts - Amherst

• Rachel R. Kachchaf, Smarter Balanced

• Ann Rosebery, TERC