Sampling Variability in the Generalizability of Observation Instruments

• In Event: 29.082 - Roundtable Session 9
  In Roundtable Session: 29.082-13 - Improving the Accuracy of Human Rating and Scoring

Abstract

Generalizability theory is being used widely to explore the measurement properties of observation protocols. However, data from observation protocols is nested, partially crossed, often unbalanced, and has low cell counts. This complex data structure is difficult to fully model. No research exists, though, to examine model uncertainty in this context. This study uses sampling across districts to estimate model uncertainty in Generalizability models applied to observation instruments. I also examine the effects of including only a subset of error facets and assuming normally distributed error facets. I show that model uncertainty can be surprisingly high, especially in a decision study context. This highlight the need to carefully consider model uncertainty in when using Generalizability theory.

Author

• Mark White, University of Michigan - Ann Arbor