Cross-Classified Random Item Effects Bifactor Graded Response Models

• In Event: Addressing Practical Issues in Measurement Invariance Research: Methodological Investigations and Advances

Abstract

Objectives
The purpose of this study was to develop a cross-classified random item effects bifactor graded response model and illustrate its use and utility in a case study of measuring teaching quality.

Framework
An important assumption underlying meaningful comparisons of scores in rater-mediated assessments is that measurement is commensurate across raters. When raters differentially apply the standards established by an instrument, scores from different raters are on fundamentally different scales and no longer preserve a common meaning and basis for comparison. Left untreated, measurement nonvariance has the potential to unfairly affect outcomes and undermine the reliability and validity of rater-mediated assessments.

Context
We applied the model to a study of teaching quality using repeated classroom observations of teaching quality with two instruments such that we considered a general factor that spanned instruments and secondary instrument-specific factors. Observations are cross-classified among raters and teachers because each observation was concurrently nested within a teacher and within an independent rater.

Methods
The model was
(1)
Here Φ is the normal cumulative distribution function, Yiotr is the score for item i in observation o for teacher t rated by rater r, Gt represents teacher t's persistent level of general quality, is the discrimination parameter for item i on the general dimension rated by rater r and the aigr have a distribution across raters such that . Further got is the general quality deviation specific to observation o for teacher t, and Mt and Ct represent teacher t's persistent level of quality specific to the Mathematical Quality of Instruction instrument (M) and Classroom Assessment Scoring System instrument (C), with m and c as their observation-specific counterparts. aimr and aicr are the discrimination parameters for item i on the instrument respective dimensions with amr and acr such that and across raters. Let K represent the number of categories items are graded on with k as a specific category and let ,..., be a set of K-1 ordered item difficulty thresholds where each has a distribution across raters such that . Dimensions were assumed to be normally distributed and orthogonal and we fixed the scale so that teacher level dimensions had unity variance. Parameter estimates were obtained using Bayesian estimation and noninformative priors (Asparouhov & Muthen, 2012; Gelman, Carlin, Stern & Rubin, 2004).

Results
The results indicated that there was significant variability in the item parameters across raters suggesting both metric and scalar noninvariance. Adjustment for this noninvariance through the proposed model improved relative model fit and the correspondence between the teaching scores and teachers’ value-added scores. Further, the results suggested that the multidimensional nature of the random item effects allows items to borrow strength from other items to improve the estimation as compared to approaches that ignore the correlations among random item effects.
Significance
The comparability of scores assigned by different raters is a well-known and complex problem. Evidence from this study suggests the promise of the proposed method in analytically addressing measurement non-invariance in rater-mediated assessments.

Authors

• Benjamin Kelcey, University of Cincinnati

• Shanshan Wang, University of Cincinnati