Putting the Pieces Together to Create a Defensible Blueprint

• In Event: Developing a Blueprint for a National Medical Licensing Examination Program: From Soup to Nuts

Abstract

Objective

The objective of this presentation is to describe the process by which a panel of subject matter experts (SMEs) integrated data sources of our practice analysis to develop a defensible blueprint for our medical licensure examination program.

Perspective

The aim of our blueprint is to ensure that the critical core competencies, knowledge, skills and behaviors required of a physician entering supervised and unsupervised practice are being fairly and consistently assessed across all test forms.

In order to better inform this process, the following sources were presented to our SMEs for review: (1) a study on resident entrustable professional activities; (2) the results of a national survey of physicians focused on the importance of knowledge, skills and attitudes for diagnostic and management activities; (3) a study on the incidence and prevalence of diseases encountered by Canadian physicians in practice; (4) the results of a survey which was centered on public expectations of their physicians.

Methods

A panel of 12 physicians (the SMEs) was asked to develop a blueprint that would be applicable for assessments leading up to two decision points, i.e. (1) entry into supervised practice residency) as well as entry into unsupervised independent) practice. In terms of composition, they were representative of all stakeholder groups and included medical regulatory authority members, undergraduate and postgraduate deans, specialty certification board members, test committee members, etc.

Results

Following a presentation of the four data sources, SMEs were asked to provide characteristics that they felt were reflective of a candidate entering both supervised (decision point 1) and unsupervised (decision point 2) practice along the following five dimensions: (1) knowledge-related; (2) skills-related; (3) behavior-related; (4) personal qualities and; (5) job related.

Subsequent to this definitional phase, our SMEs integrated the various sources of information to arrive at a blueprint that they felt not only adequately reflected the broad themes emerging from the four data sources but also resonated with their professional experiences and expectations. The outcome of this exercise was a two-dimensional blueprint defined by the following axes: (1) Dimensions of Care (Health Promotion and Illness Prevention, Acute Care, Chronic Care, and Psychosocial Aspects of Care) and (2) Physician Activity (Assessment/Diagnosis, Management, Communication, and Professional Behaviors). Clear definitions, substantiated with observable behaviors, were provided by our SMEs for these eight broad categories.

The final phase of our exercise entailed asking our SMEs to provide broad importance ratings (in the form of percentages) for each of the blueprint dimensional categories outlined above, again as a function of both decision points.

Generally, findings suggest a slightly greater emphasis on communication and professionalism than has historically been the case for candidates entering supervised practice and unsurprisingly, a stronger onus on management skills for those candidates at decision point 2.

Scholarly significance

The present study provides a clear, defensible process by which a blueprint can not only be developed, but also defended to candidates, practicing physicians as well as the public. Next steps and implications of our blueprint and test specifications for future activity will be discussed.

Author

• Andre F. De Champlain, Medical Council of Canada