Comparing the Performance of English Language Learners and Non-English Language Learners on Test Accommodations: A New Index to Evaluate Differential Boost

• In Event: Validity of Accommodations for English Language Learners in NAEP

Abstract

Objectives
There is a need for a single effect size to measure how much more effective an accommodation is for ELLs as compared with non-ELLs. The goals here are to define and apply a new effect size index to fill this need.

Theoretical Framework
Sireci et al. (2005) argued that an accommodation can be effective when there is evidence of a “differential boost” even if the gain for non-ELLs is not zero as long as the gains are significantly larger for ELLs. However, existing meta-analyses of experimental studies on accommodations for ELLs (Kieffer et al., 2009; Pennock-Roman & Rivera, in press) have not measured the amount of differential boost directly. Instead, two indices were calculated, one for ELLs and another non-ELLs. Each index calculated was a standardized difference in mean test performance between the A group--- randomly assigned to the accommodated version of a test, and the C group---one randomly assigned to the control or unaccommodated version of the same test. The indices for these groups on the same accommodation cannot be directly subtracted in a meaningful way to measure differential boost because the two indices are in different units. For example, suppose that the mean difference for ELLs was the same as for non-ELLs in the raw score metric—say +10 points. If the SDC for ELLs was 16 and the SDC for non-ELLs was 20, the standardized mean differences would be 10/16 = 0.625 for ELLs and 10/20 = 0.500 for non-ELLs. The effect size would appear 0.125 larger in the ELL group although the improvements were equal.

Method
The new index we are proposing takes the form:

[(MAF - MCF) – (MAR - MCR)]/ SDCR

Here the A and C subscripts identify means from the accommodated and control versions of tests. The F ad R subscripts identify the focal group (ELLs) and the reference group (non-ELLS). The many advantages of this index within the present context (Becker, 1988; Glass et al., 1981; Morris & DeShon, 2002) will be discussed.
Data Sources
For each experimental study in the meta-analysis of Pennock-Roman and Rivera (in press) having an effect size for both ELLs and non-ELLs on the same accommodation, an index of differential boost was calculated. The majority of studies in this analysis involved instruments constructed from NAEP test items.
Results
In some instances, the differential boost index gave quite different results as compared with the original individual effect sizes for ELLs and non-ELLs.
Significance
Investigations on the degree to which test accommodations provide a differential boost for focal groups (ELLs or students with disabilities--SWDs) versus reference groups (non-ELLs or non-SWDs) has been hampered by the lack of an appropriate effect size. The index proposed and illustrated here fills this gap. It is useful in many instances when accommodations do not maintain score equivalency for the reference group. A subsequent application of psychometric equating techniques would ensure that effective accommodations have scales equal to the unaccommodated test, thus allowing pooling of unaccommodated and accommodated scores for reporting purposes.

Authors

• Maria J. Pennock-Roman, MPR Psychometric and Statistical Research Consulting

• Charlene Rivera, The George Washington University