Building Competence in Performance Assessment: What a Faculty Needs to Know and Do

• In Event: Assessing Competence in Professional Performance Across Disciplines and Professions

Abstract

This presentation outlines components of a faculty development program in performance assessment, based on a review of the literature and the experiences of persons who have conducted courses for faculty development in performance assessment. In Tekian’s (2012) vice-presidential address to Division I: Education in the Professions, he elaborated his definition of faculty development, and then presented chronologically what was achieved and published in the literature from the 1970s until 2012. He then focused on whether and how “development” occurred during faculty development (see also Friedman, Cilliers, Tekian, & Norcini, in press). This presentation also benefits from the experiences of individuals who have conducted courses for faculty development in performance assessment. Included are basic principles, methods, guidelines for blueprinting and construction of assessments, assessor training, and scoring and standard setting. Not every faculty member needs to be competent in all areas, but some of them need to be competent in each.
1. Basic principles. A faculty needs to be generally aware of the criteria for good assessment. When they understand concepts of validity or coherence, validity of consequences, reliability, and equivalence, they are better able to participate in assessment design. When they are able to weigh issues such as feasibility, educational affect, catalytic effect, and acceptability, they are better able to discuss broader issues in performance assessment.
2. Methods and their alignment with competencies. Over the past decades, a number of different assessment methods have been developed. This section surveys methods and identifies the competencies they are best suited to assess. Included are MCQs, OSPEs, simulations, and workplace-based assessments.
3. Blueprinting and construction of assessments. In the professions, the quality of individual performance assessments is driven mainly by content integrated with skills. Consequently, it is important to systematically sample from the domains of the competencies being assessed. Likewise, the quality of assessment tasks and materials is paramount. This section includes blueprint development, job analysis, task writing, and development of stations of practical and clinical examinations.
4. Assessor training. Many of the methods of assessments used in the professions rely on the judgments of a faculty and other observers. The quality of these assessments is strongly influenced by the accuracy, consistency, and competence of the judges. This section focuses on how to train these assessors and includes practical methods for enhancing, improving, calibrating, and ensuring the quality of assessor judgments.
5. Scoring and standard setting. In many performance assessment situations, faculty judges may be assigning scores to student performances, and may make pass/ fail decisions. This section describes commonly used scoring strategies. It also highlights several methods for making pass/ fail decisions, i.e., setting criterion-based standards.
Ara Tekian and John J. Norcini provide practical examples for the five areas.

Authors

• Ara Tekian, University of Illinois at Chicago

• John J. Norcini, Foundation for Advancement of International Medical Education and Research