Construct Validity of a Grade 6 Standardized Reading Assessment Across Language Backgrounds

• In Event: 77.030 - French as a Second Language Programs in Canada: Perception, Assessment, and Academic Outcomes

Abstract

Objectives
With growing diversity among student populations, educators are increasingly using standardized literacy measures to evaluate the achievement of students from diverse linguistic backgrounds. Meaningful interpretation of results, however, requires sufficient construct validity. The goal of the current study was to evaluate the construct validity of a large-scale Grade 6 standardized reading assessment by analysing its measurement invariance between monolingual and bilingual home language speaking groups.

Theoretical framework
Using large-scale standardized assessments for accountability purposes requires valid score interpretations and appropriate test score use across diverse student populations (Niehaus & Adelson, 2013). Stability in measurement across groups must be examined to ensure equivalent construct validity. Measurement invariance is a psychometric property indicating that the underlying construct holds the same for different subgroups (Pendergast, Embse, Kilgus & Ekland, 2017). Research on standardized tests often finds a lack of measurement invariance for bilingual students (Abedi, 2002), providing a rationale for the current study. In addressing these issues, measurement invariance was examined through multi-group confirmatory factor analysis (CFA).
Data sources
This study used a provincially mandated Grade-6 reading achievement test that contained item responses from monolingual (n=19648) and bilingual (n=8979) students. The standardized reading test consisted of 26 multiple choice and 10 open response questions relating to 4 reading passages. Monolingual/bilingual status, used as the grouping variable, was determined based on student background survey data. Participants were coded as monolingual if they reported hearing and speaking only English at home, and bilingual if they reported both hearing and speaking another language as often, or more than, English.

Methods
A 2-factor multi-group CFA was used to analyze measurement invariance across groups. In multi-group CFA, measurement invariance is tested by creating a baseline model and imposing increasingly restrictive equality constraints across subsequent models (Byrne, 2013). A significant decrease in model fit demonstrates variance across groups. The planned steps in this study included testing configural, metric, scalar, and factor mean invariance; this involves the invariance of factor structure, factor loadings, item thresholds, and latent factor means respectively.

Results
Results found acceptable measurement invariance at the configural and metric level, but a significant drop in model fit when testing scalar invariance (corrected ΔMLχ2(16)= 66.23, p < .001). Violation of scalar invariance indicates that monolingual and bilingual student groups matched on the same ability level failed to have equal values on several items. Open-response items in particular tended to demonstrate this characteristic.

Scholarly significance
The current study contributes to the literature by noting the potential for measurement variance in reading assessments for bilingual students. Specifically, it highlights the importance of ensuring measurement invariance assumptions are met before using assessments to compare performance across subgroups. The trend of decreased invariance in open-response items also improves our awareness regarding the role of question format on invariance. These findings can be applied to improve the interpretation quality of current reading assessments, and promote the development of assessments that remain valid across diverse populations.

Authors

• Christine Marie Barron, OISE/University of Toronto

• Eunice Eunhee Jang, University of Toronto