Using Formative Assessment in a Medical Professional Recertification Program

• In Event: Opportunities for Formative Assessment Across Health Professions Education Settings

Abstract

In the medical profession, certification examinations serve primarily as summative assessments to decide if physicians meet the standards of competence to practice medicine after residency. The individual score report is usually the only feedback test takers receive, containing limited information regarding the content of the examination. It is not easy to identify the strengths and weaknesses of one’s performance due to the limitations of the data. Once certified, physicians, especially those who are solo practitioners, no longer have an objective and independent entity to provide formative assessment needed for their continued education.

In response to physicians’ requests for more formative feedback and a more flexible assessment experience, in 2017 and 2018 our medical board piloted an innovative assessment program to provide formative assessment to prompt learning, and also fulfill the role of a summative high-stakes assessment to recertify physicians. After the pilot, physicians may choose the new option instead of a proctored examination administered once every ten years at a secure testing center.

In the pilot, physicians received 20 time-limited (5 minutes) multiple-choice questions per quarter per year. These questions assessed the application of fundamental knowledge and guidelines used in everyday practice. They could answer questions at any time, anywhere, on any device (personal computer, iPad, or smartphone) during the quarter and receive immediate feedback: whether an answer was correct or incorrect, references supporting the correct answer, and a brief explanation of the correct answer. They could use resources such as books or the internet to answer the questions but could not consult with other physicians. After answering each question, but before they were shown the correct answer, they were asked to rate the relevance of the question to their practice, and their confidence in their answer.

In the 2018 pilot, 5,910 physicians enrolled in the program to fulfill their high-stakes recertification requirements and received a pass/fail decision at the end of the year. If they failed, they had to take the proctored secure exam at the testing center to maintain their certificate. These physicians were in the program for both formative and summative assessments. There were also 3,545 physicians who participated in the program in 2018 to earn extra self-assessment credits without receiving a pass/fail decision. This group of physicians was only in the program for the low-stakes formative component of the program. Our study will investigate whether the two groups behaved differently while participating in the same program:

· How much time did each group spend on the questions?
· Was there a difference in the pattern of taking questions in terms of batching or procrastinating?
· Was there a difference in performance on the exam?
· Did the two groups rate the same confidence level when answering the questions?
The new program will show us how the formative assessment and summative assessment complement each other, and whether test takers behave differently when taking it as a pure formative assessment or as both formative and summative assessment.

Authors

• Cathleen Koenig, American Board of Pediatrics

• Ying Du, American Board of Pediatrics

• Amanda Clauser, National Board of Medical Examiners