Estimating Interrater Reliability Using Latent Variable Modeling and Incomplete Data

• In Event: 60.030 - Psychometric Modeling of Raters

Abstract

Objectives and Purpose: In large scale assessment programs that utilize human raters, such as NAEP, the scoring design implemented will influence which measures of interrater reliability (IRR) are used. Correlation-based estimates, for example, require that two raters score students’ performances. In NAEP, a majority of student performances are scored by a single rater. IRR is estimated based on 25% of the performances being double scored.

Newer methods for estimation of IRR may reduce the need for double scoring and make scoring less resource-intensive. When a subset of raters scores student performances, it creates an incomplete dataset because a complete dataset would require that every student performance be scored by every rater.

The guiding research questions for this study are:
1) Can full information maximum likelihood estimation accurately recover IRR in scoring designs producing incomplete data?
2)Is double scoring necessary for estimating latent IRR? If so, what percentage of performances need to be double scored to accurately estimate IRR?

The purpose of this study is to determine the extent to which a latent variable modeling framework can accurately recover the IRR of scoring designs that produce incomplete data.
This study will report the results of a Monte Carlo simulation in which datasets are generated from populations with known structures.

Theoretical Framework: Under the latent variable modeling approach, the relationship between the scores provided by each rater are explained by the raters’ ability to correctly recognize the underlying construct of interest. Raters who have ability to recognize the underlying construct in student performance are theorized to provide scores that are more consistent with each other. From Raykov, Dimitrov, von Eye, and Marcoulides (2013), the model can be expressed as:

y^*=Λη+ε, [1]

where y* is the underlying student ability being evaluated by raters, Λ is the matrix of factor loadings, η is the factor, and ε is matrix of unique factors. In order to estimate latent IRR, we will fit a confirmatory factor analytic model to the observed data in order to obtain parameter estimates for the loading matrix (Λ) and error matrix (ε). These values are used to calculate latent IRR as:

ρ = [∑(λj)^2] / [∑(λj)^2 + ∑(εj^2)],

where j refers to the raters.

Data Sources: The data generated for this study uses the latent IRR model expressed in Equation 1. The design factors include sample size (100, 250, 500, 1000), number of raters (4, 6, 8), number of scale categories (4, 6, 4 with augmentation [i.e., 12], 6 with augmentation [i.e., 18]), true reliability (.7, .8, .9), percentage of double-scored performances (0%, 10%, 25%), and IRR estimators (latent reliability, G coefficient). Thus, this study employs a fully crossed design with 864 cells. We will generate 1000 datasets for each cell.

Results: The outcome of interest is the accuracy of IRR when estimated with incomplete data. We measure accuracy using the difference between the true and estimated IRR in each dataset and report aggregated results. We anticipate the completion of this Monte Carlo investigation no later than December 2014.

Authors

• Grant B. Morgan, Baylor University

• Robert L. Johnson, University of South Carolina

• Kari Hodge, NACE International Institute