Use of Affective Measures in the Training Assessment System

• In Event: New, Generalizable Strategies for Developing Assessments for the Naval Education and Training Command

Abstract

One of the Navy stakeholders’ requirements for the new Training Assessment System was to assess the affective or non-cognitive impact of the system. The two affective variables that CRESST implemented for this requirement were self-efficacy and anxiety. This paper will report empirical data on reliability and validity inferences for these measures. In addition, the impact of the system instantiation on these constructs will be described.

The goal of this system was to be the governing framework for the development of a new Navy assessment system and to test its utility by developing assessments and fielding them in a trial to assess their quality and utility. These three different assessment formats are comprehensive items, performance assessments, and knowledge maps, in addition to affective scales assessing self-efficacy and state worry. These items are state items, so trainees may react differently to the item types (that is, comprehensive items vs. performance assessment tasks).

A total of 55 trainees (4 female, 51 male) in Fire Controlmen (FC) A-school and 11 trainees (1 female, 10 male) in Damage Controlmen (DC) A-school participated. In addition, four FC instructors and three DC instructors also completed the assessments. A 10-item affective measure querying state self-efficacy (5 items) and state worry (5 items) (adapted from Mulkey, 1996; AUTHOR, 1999) was administered twice, once after the comprehensive items and again after the performance assessment. Self-efficacy items and state worry items alternated and appeared in the same order for all trainees.

The data was obtained in a formative evaluation of the assessment system. Assessments were operationally delivered on iPads. Data from the sampled trainees responding on a pre- and post-assessment basis were analyzed to report descriptive differences among groups and between courses by type of assessment.

Table 1 describes statistics of the FC cognitive and affectiveness measures. As may be seen in Table 1, self-efficacy was low and anxiety was moderate, consistent with a testing environment. The alpha reliabilities were considered appropriate.

The correlations among the affective measures for the FC rating indicate that there were no significant correlations between either affective measure and comprehensive test items. However, FC Performance Assessments 1 and 3 were significantly correlated with self-efficacy.

As may be seen in Table 2, self-efficacy was higher for this rating and anxiety was somewhat less. Table 2 shows descriptive statistics of the DC cognitive and affective measures.

Unexpectedly, there were no significant correlations between either affective measure and comprehensive or performance test items.

The analysis indicates that our general approach to the measurement of affect as an outcome variable has merit in formative evaluation. Our measures for affective measures could be used by other researchers and help to advance the state of the art in this area. This study was driven by stakeholders’ requirements.

Author

• Harold F. O'Neil, University of Southern California