Is Programmatic Assessment a Practically Useful Theory of Assessment in Competency-Based Education?

• In Event: Assessment Issues in Competency-Based Education in the Professions

Abstract

Perspective: Over the past few decades, the competency-based movement has been driving change in education for the professions. Competency-based education (CBE) calls for a fundamental shift in our thinking about learning and expertise development, curricular models as well as assessment practices. The implementation of competency-based assessment has been proven difficult (Holmboe and Batalden, 2015). In fact, the competency-movement has been critiqued for its approaches to assessment. More specifically, it is being argued that approaches to measurement of outcomes in CBE interfere with development of trainees into self-directed, effective learners (Brooks, 2009). Recently proposed models of Programmatic Assessment (PA) aim to address these problems in CBE by integrating and optimizing the learning (formative) and decision making (summative) functions of assessment. PA builds on well-researched assessment principles (Van der Vleuten et al., 2010). Key principles underlying programmatic assessment are:
- The traditional dichotomy between formative and summative assessment is redefined as a continuum of stakes, ranging from low- to high-stakes decisions;
- Individual assessment occasions are optimized for providing feedback to the learner, not for pass-fail decisions;
- Pass-fail decisions are made on the basis of many, aggregated data and procedural measures to ensure trustworthiness of judgments.
Objective: This paper presents and analyzes 2 ‘best practices’ of PA in different contexts in medical training. We aim to refine assessment theory and facilitate implementation of PA in various educational settings through identification of context-specific factors for success and issues of concern.
Methods and Data Sources: A small-scale preclinical program (50 students/year) and a large-scale clinical rotations program (360 students/year; 10 hospitals) at Maastricht Medical School. We use formal evaluation reports, interviews and surveys from students and staff to evaluate utility of PA-approaches. We focus on criteria related to implementation and effectiveness in terms of formative and summative assessment functions.
Results: PA is feasible and acceptable, even in large scale clinical settings. PA stimulates learning, especially when the program allows for creation of small communities of learning and mentoring relationships that stimulate reflective practice and follow-up on feedback. Learning and intrinsic motivation is furthermore enhanced if students feel that they are actually able to self-direct and co-create learning and assessment processes, which is more difficult in an input-based, fixed curriculum. Learning may be hindered by overly structured feedback processes, excessive reflections as well as provision of grades and normative judgments. Documentation of feedback in a portfolio seems to result in early identification and remediation of poorly performing students in clinical settings. Main issues of concern are related to faculty buy-in in large scale settings and, more important, commitment and training of students as they appear to be driving spirits in change.
Significance: PA seems a promising approach to overcome assessment concerns in CBE. Successful implementation is dependent on curriculum structure, educational design and feedback culture in which high levels of accountability are combined with high levels of psychological safety. Supporting students in developing the habit of assessment and excellence is crucial.

Author

• Marjan Govaerts, Maastricht University