Reliability of Article-Based Testing: Insights from the Omega Coefficient Family

• In Event: Virtual Poster Session
  In Virtual iPresentation Only: Division I - Education in the Professions Virtual Poster Session

Abstract

The Article-Based Continuing Certification Pathway is the alternative to the traditional Continuing Certification exam of American Board of Psychiatry and Neurology (ABPN). For each certification, diplomates must pass a specified number of open-book exams based on journal articles every three years; passing each article exam requires answering at least four out of five questions correctly. Evaluating reliability for this novel assessment requires careful consideration, as traditional measures such as Cronbach’s alpha may underestimate/misrepresent the test’s reliability. To provide a more robust estimate of reliability for article-based assessment, this proposal explores the feasibility of Omega coefficient family, which accounts for factor loadings and multidimensional structures. Using the first three years data, we demonstrate how Omega coefficients enhance score interpretation in measurement.

Authors

• Qiao Lin, American Board of Psychiatry & Neurology

• David Shin, American Board of Psychiatry and Neurology, Inc.

• Jeffrey M. Lyness, American Board of Psychiatry and Neurology, Inc.

• Huaping Sun, The American Board of Anesthesiology