Synthesizing and Reporting Learner Analytics: Considerations for Measurement and Validity

• In Event: Big Data: A Big Sell or Oversold in Professions Education

Abstract

Assessing learners and providing meaningful feedback has been the cornerstone of training health professionals. Recent reforms in the structure of healthcare systems and mandates by accreditation agencies have accelerated the need to improve systems that combine scores from multiple assessments to promote better tracking and reporting of learner (Institute of Medicine, 2014; Nasca, Philibert, Brigham, & Flynn, 2012). In this respect, training programs have struggled to collect and report meaningful information to trainees in a systematic and real-time manner. Consequently, software-based learner analytics platforms such as dashboards have been developed and used to facilitate reporting of learner progress (Taveria-Gomes et al., 2012). However, examining the validity of these scores become increasingly challenging because training occurs in complex clinical environments with a variety of patients, healthcare providers, and availability of resources. In other words, operationalizing these assessment systems to reliably measure learner progress and also to provide meaningful and diagnostic feedback is a key issue facing program directors and educational programs (Park, Hodges, & Tekian, 2016; Park, Zar, Norcini, & Tekian, 2016). This paper presents a framework for synthesizing learner analytic data that considers both psychometric characteristics and validity of scores. Evidence from a three-year longitudinal cohort of trainees will be used to present our findings and implications.

While reporting of individual assessment scores such as an observation of patient encounters using an assessment tool can be relatively straightforward, it is unclear how best to combine scores from multiple assessments that best captures the developmental progress toward mastery in a specific competency or professional activities (Bienkowski, Feng, & Means, 2012; West, 2012). Assessments in medical education are conceptualized within a “systems” framework, where scores from multiple assessments are combined and used to inform programs (Holmboe & Hawkins, 2008). These psychometric considerations have not been incorporated sufficiently for learner analytics tools, which raises concerns on their consequential validity and impact (Steventon & Billings, 2017).

In this study, we present a framework for synthesizing multiple assessment scores by introducing a data analytics tool that provides real-time information to learners, for a three-year longitudinal cohort of 30 trainees. This approach presents an attempt to quantify and systematically integrate various assessment scores that provides a practical and reasonable solution for generating fine-grained profiles of learner progress that could be used to make decisions. Within this framework, Kane’s composite score approach (Kane & Case, 2004) is used to synthesize assessment scores that have been found to predict low-performing learners (Odds Ratio = 5.82, p < .001). This study also discusses psychometric and validity considerations on the utility of learner analytic tools and their scores. More specifically, we discuss broad implications on how well learner analytic scores can be improved to (1) predict future performance and provide early detection of low performers, (2) identify profiles of learners at competency levels that can generate useful information for program directors, and (3) yield better reliability to provide more consistent information to learners and educators. Implications for data visualization and access to real-time information will also be discussed.

Authors

• Yoon Soo Park, University of Illinois at Chicago

• Ara Tekian, University of Illinois at Chicago