Addressing the Linguistic Challenges of Assessing English Learners: A State and Research Organization Partnership

• In Event: Collaboration to Improve Educational Opportunities for English Learners in the United States

Abstract

Objectives: This poster describes the collaboration between researchers and state department of education administrators to investigate how the language of grade 5 multiple-choice state science test items may affect the ability of ELs to demonstrate what they know about science when answering the items. The state assessment is administered every year to students in grade 5, grade 8 and high school, and fulfills federal and state accountability requirements.
Perspectives: The majority of state science tests are given in English, with limited accommodations available to ELs (Wolf et al., 2008), and as a result, ELs’ test scores may not always reflect their knowledge and skills in science. Researchers have explored the specific linguistic features of science test items that interfere most with ELs’ comprehension of test item texts (Noble et al, 2014). In this collaboration, we built upon prior research to explore ways to improve the fairness of one state’s science test for ELs and the validity of interpreting ELs’ scores on that test as representations of their knowledge and skills in science.
Methods: Throughout this project, researchers and administrators have developed shared understandings of how to use emerging research findings to inform test item development. For example, administrators have discussed individual items with researchers, to uncover research-based hypotheses for the low levels of performance of ELs on the test items, and to discuss possible linguistic modifications to the test items. In addition, researchers have participated in assessment development meetings to review candidate test items for inclusion on state tests and to share research-based findings regarding linguistic features that differentially affect the performance of ELs.
State administrators and researchers collaborated on an experimental test of the hypothesis that specific linguistic features of state test items were lowering the performance of ELs in the state. Twenty-two grade 5 multiple-choice science test items were linguistically modified, while maintaining the same science content. State administrators collaborated with researchers on the development of linguistically modified items, shared state demographic and test score data critical to the study, reviewed and discussed findings with researchers, and helped to disseminate findings to participating school districts in the state.
Results. Over 2200 students (ELs and non-ELs) were given tests consisting of original and modified test items, and the findings of this study include measures of the overall effectiveness of modifications and of the effectiveness of individual modifications targeting specific linguistic features.
Significance. This collaboration has facilitated research that is informed by the larger field of assessment research, but also focused specifically on one state’s tests. As a result, the findings of this work have been directly applicable to the challenges faced by state administrators in improving their science test items.

Authors

• Tracy E. Noble, TERC

• Catherine Bowler, Massachusetts Department of Education

• Rachel R. Kachchaf, Smarter Balanced

• Ann Rosebery, TERC