Investigating the Unidimensionality Assumption in Alternate Assessments

• In Event: Issues in Large-Scale Alternate Assessments for Students With Disabilities

Abstract

1. Objectives
The purpose of this study is to examine whether alternate assessments meet the unidimensionality assumption by using real data from two state-wide alternate assessments. Compared to other K-12 assessments, it is more difficult to investigate unidimensionality in alternate assessments because of inadequate sample size and complex test design of alternate assessments. This study demonstrates different methodologies for examining unidimensionality in alternate assessments.
2. Theoretical Framework
Unidimensionality in assessments refers to a single construct being measured by the test. Most states in the US employ unidimensional item response theory (IRT) models to estimate student scores in K-12 assessments. To use unidimensional IRT models, it is very important to ensure that the unidimensionality assumption is adequately met. The violation of the unidimensionality assumption may cause psychometric issues such as test bias and miscomputation of test scores.
3. Methods
Data analysis consists of two parts. First, a one-factor confirmatory factor analysis (CFA) model is fit to math and reading data. Then, the fit of this model is evaluated based on model-fit indices. Second, a multidimensional IRT (MIRT) model is fit using the content domains as separate dimensions. The fit of this model is compared against the fit of unidimensional model. In addition, vertical scale that places different grade levels on the same score scale is examined in terms of unidimensionality using the same methods. Data from the same students across two consecutive test administrations (e.g., 5th grade in 2012 and 6th grade in 2013) are used. Two sets of student responses are treated as a single dataset and the one-factor CFA model is fit the data. Also, each year’s data are treated as a separate dimension and two-dimensional MIRT model is fit the whole data.
4. Data Sources
The data used are from two state alternate assessment programs for math and reading from grade 3 to 10.

5. Results
The initial findings indicate that there is an interaction with disability type and students’ ability levels. Students with different disability types tend to take different set of items on the test based on their initial scores. For math, one-factor CFA model results for 3rd-5th and 6th-8th grade bands indicate good fit, meaning that the test is unidimensional. In MIRT analysis, the scores based on the content domains are highly correlated, which also implies that the content domains actually define a common factor. Similarly, dimensions based on two consecutive years are highly correlated, meaning that both tests measure the same construct. The final paper will provide more in-depth discussion of the findings.
6. Significance of the Study
Although the number of students taking alternate assessments is only 1% of the whole population, alternate assessments require more attention in terms of test development and the investigation of psychometric properties of the test. The results of this study will provide insight into the extent which alternate assessments are able to measure a unidimensional construct and whether there is an interaction between disability types and the test.

Authors

• Okan Bulut, University of Alberta

• Ming Lei, American Institutes for Research