A Novel Exploration of Automating Scoring of Communication Skills

• In Event: Assessment and Feedback in Health Professions Education

Abstract

Research has shown consistently that communication skills are essential to quality medical care (e.g., Riedl & Schüßler, 2017). We sought to advance the assessment of communication skills by implementing automated scoring of transcripts obtained from virtual objective structured clinical encounters (OSCEs) with virtual standardized patients. Trained raters annotated demonstrations of correct behaviors in encounter transcripts. These demonstrations were input to a natural language processing (NLP) engine for automated scoring of communication skills. We found encouraging preliminary results for the feasibility of using automated scoring of communication skills. Moreover, our methods of expert review were able to meaningfully improve initial results with respective to false negative rates. Future research should investigate improving the false positive rates obtained using NLP.

Authors

• Andrew Houriet, National Board of Medical Examiners

• Ann King, National Board of Medical Examiners

• Jerusha J. Henderek, NBME

• Janet Mee, NBME