Identifying Examinees With Item Preknowledge: Kullback-Leibler Divergence or Posterior Mean Difference?

• In Event: 65.079 - Roundtable Session 23
  In Roundtable Session: 65.079-18 - Methods of Detecting Response Collusion and Preknowledge

Abstract

When examinees gain access to test items before an examination, (i.e., gain “item preknowledge”), the validity of inferences made from their resulting test scores is seriously diminished. Test scores tainted by the effects of item preknowledge are particularly unacceptable for use in health care licensure decisions which have serious implications for public safety. This study used simulated data to compare the performance of two methods for identifying examinees that benefit from item preknowledge – one based on the Kullback–Leibler divergence (KLD; Kullback & Leibler, 1951), and one based on the mean difference between posterior distributions. Initial results indicate the posterior mean difference (PMD) method may outperform the KLD method in terms of preknowledge detection power and type I error rates.

Authors

• Matthew Grady, American Dental Association

• Chien-Lin Yang, American Dental Association