Students Response Processes as Validity Evidence for the Accommodations for English Language Learners

• In Event: Validity of Accommodations for English Language Learners in NAEP

Abstract

This presentation describes how cognitive methods can be used to investigate student response processes to be used as empirical evidence in the development and validation of educational assessments, specifically accommodations, for English language learners.

When assessing ELL students, a central validity concern is to what extent the assessment outcomes demonstrate the students’ content knowledge and skills without interference from the students’ limited English language proficiency. To provide valid assessments for ELL students, the effects of unnecessary linguistic complexity of assessments should be mediated by either reducing the unnecessary complexity or by providing accommodations that help ELL students with their linguistic needs. Accommodations are meant to help ELL students overcome the language barriers they may face in understanding and responding to content assessments. With the appropriate use of accommodations, the validity of inferences made from the content assessments for ELL students can be improved. Federal legislation specifically urged states to establish an appropriate, valid accommodation guideline as part of their accountability systems (NCLB, 2002). However, challenges emerged partly because of inconclusive results on effective accommodations.

While accommodations are commonly used in practice, their validity and effectiveness warrants continual investigation. This presentation discusses the application of cognitive interviewing and think-aloud protocols in the validation of mathematics items and accommodations for ELL students at various levels of English Language Proficiency. Transcriptions from think-aloud protocols of ELL and Former ELL students responding to items of varied linguistic complexity will be presented to illustrate how direct linguistic accommodations can successfully address the barriers posed by excessive linguistic complexity in content assessment items for this population of students. Implications for the use of direct linguistic support accommodations for ELL students and the evaluation of the effectiveness of accommodations will be discussed.

To gather student response processes while solving mathematics items, test items were presented to ELL and Former ELL students, who read aloud the item text in English. Students explained what the item was asking them to do. Probe questions assessed whether students could a) understand the text in English; b) rephrase the text in Spanish or English to demonstrate their understanding of its meaning; c) identify which aspects (if any) of the English text they could not understand; and d) figure out what the item was requiring them to do, even if they could not understand the text in its entirety. The types of mistakes students made when interpreting linguistically complex text were recorded as well as whether they could answer the question correctly based on their understanding of the text. The presentation will present transcripts of these interviews to illustrate language comprehension problems that are common to items and how accommodations can be designed to address these language issues.

Finally, the presentation will discuss practical implications and recommendations for assessment development and for the assignment of accommodations to ELL students at different levels of English language proficiency.

Author

• Maria Martiniello, Tomas Rivera Policy Institute