Validity of Mathematics and Science Assessments of English Language Learners in Australia, Canada, the United Kingdom, and the United States

• In Event: Validity and Fairness Issues in Assessing Second-Language Test-Takers

Abstract

Research has demonstrated that proficiency in the language of testing affects students’ ability to read, understand, and respond to test questions and can be viewed as a source of measurement error (Solano-Flores, 2011). Thus, performance on the same assessment can be differentially impacted by students’ language backgrounds (Abedi, 2004; Abedi, 2008; Abedi, Hofstetter, & Lord, 2004). In line with this idea, Abedi and Lord (2001) found that reducing the linguistic complexity of mathematics items resulted in higher performance of English language learners (ELLs) and the gap in achievement between ELL and non-ELL students is largest when language demand is high (Abedi, Leon, & Mirocha , 2003). Solano-Flores (2008) emphasizes that student performance on assessments is a result of a complex interchange among factors such as the student’s home language and culture, the context of the test, linguistic demands of items, and the language preferences of test developers.

This complex relationship between student language background and their performance on assessments raises questions about the validity of inferences from such assessments. In this study, we examine the degree to which English language competency accounts for variation in students’ performance on mathematics and science assessments. This research is unique in three important ways. First, it estimates the percent of variance in mathematics and science scores accounted for by English language scores. This type of statistical analysis allows us to examine to what extent performance differences in mathematics and science for ELL and non-ELL students are associated with language competency. Second, the research is conducted from an international perspective and compares these performance differences in four English speaking countries with significant immigrant populations: Australia, Canada, the United Kingdom, and the United States. This international perspective provides opportunities to gain insights about the interactions among language, educational achievement gaps, immigration policies and populations, and societal contexts in the four countries. Third, this research is conducted using nationally representative samples from these countries, allowing generalization claims at the country level.

Using data from the PISA 2009 assessments, our results indicate that between 39% and 57% of the variance in mathematics and science scores is accounted for by language proficiency; therefore performance on mathematics and science assessments cannot be meaningfully interpreted without taking language proficiency into account. After taking language proficiency into account, differences in mathematics and science achievement favoring non-ELL students were reduced in all four countries. In Australia, the ELL and non-ELL mathematics achievement differences are reversed, with ELLs scoring higher after taking language proficiency into account. In Canada, the differences between these language groups disappeared. In the United Kingdom, the achievement differences between the two groups are reduced and are no longer statistically significant. The United States is the only country where non-ELL students continued to score statistically significantly higher than ELL students after accounting for language proficiency. The research contributes both to validity research methodology as well as to research evidence on the validity of assessments of mathematics and science knowledge taken by ELLs.

Authors

• Kadriye Ercikan, The University of British Columbia

• Yue (Michelle) Chen, The University of British Columbia

• Juliette Lyons-Thomas, Regents Research Fund

• Shawna Goodrich, The University of British Columbia

• Dallie A. Sandilands, EMP Educational Measurement Professionals

• Wolff-Michael Roth, University of Victoria

• Marielle Simon, University of Ottawa