What Features of Academic Vocabulary Test Items Make Them More Difficult for English Learners?

• In Event: The Multidimensionality of Vocabulary Knowledge: Implications for Assessment and Instruction With Multilingual Students

Abstract

Objective: In the current study, we used archival academic vocabulary test data from middle school students to determine item difficulty for groups of English learners compared to English-only (EO) students. Our objective was to identify features of items that explained differences in difficulty across groups, leading to the eventual revision and improvement of the academic vocabulary test.

Theoretical Framework: Our work has been informed by lexical processing studies that explore multiple word-level features to explain speed and accuracy in lexical decision and naming tasks. We combined these features of interest with an analytic approach that focused on estimating item difficulty, allowing a more fine-grained analysis of individual items.

Methods: Descriptive item response theory models were used to estimate item difficulties. Separate models compared performance of EO students to three separate groups of English learners: those identified as initially fluent (IF), those identified as limited English proficient (LEP), and those who were initially limited but later reclassified as fluent (RF). Then in a second stage, the item difficulty estimates obtained from these analyses were used as the outcome, and item features were included as explanatory variables.

Data Sources: The primary measure was the Word Generation Assessment of Academic Vocabulary. We analyzed data from the pre-test assessments from the Word Generation trial, from a sample of 13,780 students who attended thirteen middle schools in a large, diverse urban district in California. Item feature information was obtained from various databases: word frequency from Zeno et al. (1995); contextual diversity from Byrsbaert and New (209), and number of senses and meanings from Miller et al. (1990).

Results: Word frequency, contextual diversity, and number of senses and meanings emerged as consistently important variables. In separate models, frequency and contextual diversity were significant for EO students (higher frequency or diversity predicting easier items), with small but significant increases in slopes for IF and RF students, and significant decrease in slopes for LEP students. Number of senses and meanings was also significant for EO students (more senses and meanings predicting easier items), with slopes not significantly different for IF or RF students, but a significant decrease in slope for LEP students. In a combined model, only contextual diversity remained as a significant unique predictor of item difficulty.

Significance: Our findings confirm the advantage for higher-frequency words on academic vocabulary tests, and show that frequency is less predictive of performance for LEP students than it is for others. In contrast, students who are English learners but who are proficient in English appear to benefit as much or more from higher-frequency words as English-only students. We found a similar pattern of results for contextual diversity and number of meanings and senses, suggesting that these variables should be considered further by researchers and test developers. These findings will enable us to refine the existing academic vocabulary test to contain fewer items that function differentially for students with limited English proficiency.

Authors

• Autumn McIlraith, University of Houston

• Paulina Kulesz, University of Houston Times

• Joshua Fahey Lawrence, University of Oslo

• Rebecca E Knoph, University of Oslo

• David J. Francis, University of Houston