Using Artificial Intelligence to Aid Feedback and Employee Performance

• In Event: Changing Trends in Assessments Across the Health Professions: Applications of Artificial Intelligence and Machine Learning

Abstract

Health education programs often seek tools to enhance skill assessment such as identifying performance indicators and providing better feedback. Yet, doing this well and frequently is labor intensive. AI has been shown to categorize information by identifying patterns and trends, coding information, and generating reports. 1,2 Thus, utilizing artificial intelligence (AI) can be one way to provide feedback efficiently and effectively. This study highlights its use in providing feedback to faculty and staff. Utilizing AI to sort data to identify trends and patterns may be helpful to many fields.3,4 This can be used for customer feedback, identifying risk factors, and providing personalized support.1,3 While there are concerns about AI usage and some process concerns must be mitigated5, it also addresses challenges in providing timely helpful assessments.
This presentation examines how AI can be best used to process feedback about clinical sites. Authors organized key information from survey responses to utilize AI to deliver timely feedback and a summative overview. It categorizes comments, identifies themes, and summarizes information according to request. Reports then summarize positive and negative themes and suggestions for improvement. This system has the potential to significantly improve the quality of clinical practice sites. It also provides clinical year mentors with information to discuss with students as well as ways to improve their own performance. In addition, major themes can be reviewed with students to address matters early in their clinical experience. The themes and summaries were compared to human categorizations and summaries to test for consistency and how to improve AI output.
The presentation discusses the specific technical details of the system, as well as the results of our experiments. We also discuss the potential benefits and challenges of using AI to provide feedback to health professionals. We found that some techniques work best to train the system by breaking down information further and through grouping data to receive better data faster. The system was able to sort negative and positive themes. It was also able to categorize comments well. The themes identified were consistent with humans' categorization. It was also able to write summaries of different topics. While the summaries were aligned with comments and categorization, humans did a better job with some areas and in providing nuances. We also note concerns regarding areas of data quality and interpretability. However, overall, the categorization of positive and negative themes and the brief summary of the themes were excellent for providing timely feedback draft reports and identifying concerns which should be addressed quickly. There is reason to promote AI in qualitative research6, however, the challenges to adopting AI still need to be mitigated. Therefore, we are still in need of a human perspective in analyzing data; yet AI can help process information more efficiently.

Authors

• Kadian McIntosh, University of Arizona

• Shripad Sinari, University of Arizona