Exploring the Relationship between Rater Effect Indicators from the Many-Facet Rasch Model and Network Analysis with Incomplete Scoring Designs

• In Event: Methodological Considerations for Rasch Measurement Research

Abstract

Researchers are frequently concerned with rater effects (e.g., errors and biases) in rater-mediated assessments. Methods for detecting rater effects with the Many-Facet Rasch Model (MFRM) are well established. Researchers have recently emphasized alignment between MFRM and Network Analysis (NA) rater effect indicators. NA techniques are promising in this context because they offer detailed insight into relationships between raters. However, existing NA research on rater effects relies on complete data with a limited range of characteristics. We used a real data illustration and simulation study to explore the correspondence between MFRM and NA rater effect indicators. We found close alignment between MFRM and NA rater severity indicators, but weak relationships for rater inaccuracy. We consider implications for research and practice.

Authors

• Stefanie A. Wind, University of Alabama

• Daniel O. Oyeniran, University of Alabama