Measurement Considerations for Formative and Longitudinal Assessments: Revisiting, Rethinking, and Reimaging Our Approaches

• In Event: Changing Trends in Assessments Across the Health Professions: Applications of Artificial Intelligence and Machine Learning

Abstract

Competency-based medical education (CBME) organizes curricula and assessment practices around a predetermined set of capabilities across multiple aspects of professional practice (rather than just knowledge) (Frank et al., 2010; Holmboe et al., 2018; ten Cate, 2017). It also employs a progressive framework where learners advance at their own pace by demonstrating the level of competence appropriate for the stage of their education or training. Within CBME, the central purpose of assessment is to facilitate learning in support of the developmental progression of physician competence (Holmboe et al., 2010; van der Vleuten et al., 2012). Not surprisingly, CBME implementation coincides with increased attention to learning-focused assessments based on direct observations in the clinical environment over time and fewer high-stakes examinations of practicing physicians. This translates to an emphasis on longitudinal performance assessments and a move away from traditional single-point-in-time multiple-choice question (MCQ) examinations. Such a move requires us to rethink the use of traditional psychometric practices in developing, administering, and evaluating assessments in medicine education and practice.
To make comparisons across multiple assessment points, longitudinal assessments need to ensure that repeated measures reflect the same construct on the same scale or that there is longitudinal measurement invariance (Keefer et al., 2013; Meade, et al., 2005). At the same time, constructs need to be revisited over time such that they remain relevant to specific time periods and assessments need to be suited to the developmental level of the learner (Keefer et al., 2013; Meade, et al., 2005). These seemingly contradictory elements raise important validity questions unique to longitudinal assessment designs. Determining the appropriate scoring mechanisms and feedback practices represents another area where existing models may need to be reconsidered. Different analytical approaches will be appropriate when a change in competence is measured as opposed to competence at a single time. Moreover, how best to provide a single score, if a single score is deemed appropriate, needs to be fleshed out since in some cases summing or averaging across assessment occasions may not be useful (Gorter et al., 2015). Lastly, standard setting procedures that have been used historically may not apply in assessments with multiple measurement occasions over time and/or formative assessments.
In practice contexts, many credentialing bodies in medicine have moved to a longitudinal assessment model. The dual audiences for credentialing – test takers and the public -- are rethinking high-stakes credentialing assessments that are accountable for measuring fitness and competency (to protect the public) as well as maintaining practice relevance and efficiency (for test takers). Balancing these dual demands has created an increasingly fruitful space to explore some of the empirical issues related to making high-stakes decisions on information gathered across time, while individuals’ abilities are growing and changing, and while demands on practitioners are evolving.
We review and consider traditional measurement concepts and practices in light of moves toward formative, longitudinal assessments in medical education and practice. We discuss the measurement challenges and opportunities associated with these trends, raising questions about best practices and highlighting areas of promise for both assessments of learning and assessments for learning.

Authors

• Amanda Clauser, National Board of Medical Examiners

• Monica M. Cuddy, NBME