Training Assessment Framework System

• In Event: New, Generalizable Strategies for Developing Assessments for the Naval Education and Training Command

Abstract

This paper will describe the underlying approach to the development of TAF assessment system, intended for use in initial schooling (with over 60 career specific courses) overseen by the Navy Education and Training Command (NETC). The model of assessment development guiding our practice is based on coherent, consistent ruleset (AUTHOR, 2018) useful across a variety of assessment purposes. This paper will share the approach and the data obtained over this multi-year activity.

TAF is grounded in a rule-based model of assessment design partially encompassed in the IEEE Document (ISO/IEC/IEEE, 2011). First, the views of stakeholders are prominent and define purposes. Their continuing interactions directly or through proxies contribute to assure a usable, valid, and economical system. TAF starts with the stakeholders, from the highest levels in the Office of Secretary of Defense through Navy leadership to whom NETC reports. In addition, NETC management, instructional leadership, instructional providers, and sailors are involved. NETC is supported by the Office of Naval Research to review science and technology and by the Operational Fleet. TAF posits sets of rules and boundaries around assessment development featuring domains of skills, e.g., problem solving, content, electronic circuity, and families of tasks instead of broadly defined constructs.

Assessments have been developed for end-of-course use for three different courses (electronics, ship protection, and personnel functions). Within each, assessments have been designed using three different task formats: comprehensive examinations testing knowledge and theory; performance assessments, assessing problem solving and procedural learning; and knowledge mapping (AUTHOR, 2012) to show sailor’s understanding of the system. These assessments are subjected to utilization trials, confirmation testing, and analyzed for reliability and validity. They are also revised as needed.

Pre- and post-measures for targeted Navy courses were collected. Assessments were presented on iPads for ease of monitoring and data transfer. Multiple performance assessments provided scenario-based problems and a range of response options. A common structure was used across courses to attempt to discern elements of comparability in addition to domain specific material.

Data were collected from 311 different encounters with NETC trainees, preceded by review and suggestions from subject matter experts (SMEs) on importance, frequency, and clarity of tasks. We have documented common patterns of success across the three task types, and shown areas of difficulty for sailors and SMEs who completed the examination. Details of technical quality and validity will be in a compatible presentation. Faced with limited time, constrained access to respondents, and multiple courses, we also invented better tools to support our development and analysis.

Assessment development has largely proceeded using broad constructs whether or not tied to statements of standards or goals. The Navy needs more specific yet complex skills to be measured and is unable to routinely generate usable, valid items connect to constructs. For this reason, as well as the challenges of comparability among disparate courses and the development and scoring routines of modular, scenario-based performance assessments, the work of this project has scholarly and practical meaning for workplace assessment, performance assessment, and generalizability of skills.

Authors

• Eva L. Baker, University of California - Los Angeles

• Jenny C. Kao, University of California - Los Angeles

• Stephen C Court, UCLA