Modeling Directional Testlet Effects on Multiple Open-Ended Questions

• In Event: New Developments in Item Response Theory Modeling and Applications

Abstract

Directional testlet effect (DTE) refers to a recursive influence whereby responses to earlier items can positively or negatively affect the scores on later items in the same testlet. This study aims to introduce an innovative measurement model to describe DTEs among multiple polytomously scored items. Through simulations, we found that (1) item and DTE parameters can be accurately recovered in Latent GOLD®, (2) ignoring positive (or negative) DTEs by fitting a standard item response theory model can result in the overestimation (or underestimation) of test reliability, (3) collapsing multiple items of a testlet into a super item is still effective in eliminating DTEs, and (4) fitting the existing random-effect model fails to adequately account for DTE.

Author

• Kuan-Yu Jin, Hong Kong Examinations and Assessment Authority