A Study of Differential Item Functioning in Large-Scale Alternate Assessments

• In Event: Issues in Large-Scale Alternate Assessments for Students With Disabilities

Abstract

Objective

The purpose of the study is to explore the common features of items that are identified with sever DIF. This is not a study about the statistical methods in DIF analysis.

Perspectives

Statewide large scale alternate assessments have been developed and used for students with significant cognitive disabilities. To ensure the fairness of an assessment, items should behave the same across different groups of students possessing the same ability.

DIF occurs when students from different groups with the same underlying ability have a different probability in providing the same response to an item. An item with DIF is not necessarily biased. Other factors, such as instructional differences, can also cause DIF.

DIF studies are usually conducted between gender and among ethnicity groups in assessments for students without disabilities. In alternate assessments however, since approximately 80 – 90% of the students are either moderately to severely mentally disabled or display types of autism, it can be considered even more important to assess DIF among students with different types and levels of mental disabilities.

Method
In the study, DIF is assessed between the following focal and reference groups: students with moderate and severe mental disability vs. students without severe mental disability, and students with autism vs. students without autism.

Since most items are polytomous, the generalized Mantel-Hansel (MH) method along with the standardized mean difference (SMD) (Dorans, Schmitt & Bleistein, 1992) are used to identify items displaying DIF. When the PMH < 0.05 and the |SMD/SD| > 0.25, the item is categorized as severe DIF.

Items with severe DIF will be further investigated about their types, contents, formats, presentations, and response options. Findings will be summarized across the three states. Common features among the items displaying DIF will be extracted and discussed if found to be present.


Data

The data used come from three state-wide operational alternative assessment programs used for accountability reporting from 2009 to 2012. Four subjects (math, reading, science, and social studies) are involved. Owing to state variations, not all states have all four subjects nor do all assessments have operational data for all four years, although all three states test reading and mathematics at least yearly.


Results

Initial results from the study show that there are not many items displaying severe DIF. More detailed results will be presented in the final paper.

Significance of the Study

So far, most DIF research aims at comparing statistical methods using data from the assessments for general students. Few studies have explored the common features of the DIF items used in alternate assessments. The authors believe that the findings will improve item writing for alternate assessments, and hopefully classroom instructions for students with mental disabilities.

Reference
Dorans, N. J., Schmitt, A. P., & Bleistein, C.A. (1992). The Standardization Approach to Assessing Comprehensive Differential Item Functioning. Journal of Educational Measurement, 29:309‒19.

Authors

• Ming Lei, American Institutes for Research

• Rong Jin, American Institutes for Research