Assessing Performance in Engineering Education: Examples Across 20 Years of Practice

• In Event: Assessing Competence in Professional Performance Across Disciplines and Professions

Abstract

Following Furneaux (1962), who factor analyzed six student written exams and found a single factor, course designers created a new course in engineering science at the curriculum’s advanced level. Written exams had been based on what designers thought engineers did, but couldn’t assess everything of value. For creativity, design, project planning and evaluation, and investigating skills, assessment designers introduced a non-traditional approach. Each student would submit reports on four discovery investigations (later reduced to two) that took no more than 12 hours, and one project of 50 laboratory hours. Each assessment was competency- based and criterion-referenced.
However, assessors found rubrics difficult to use. Answering ‘yes’ or ‘no’ didn’t allow for gradations. We conducted an experiment, asking students to determine electrical component(s) between two terminals of a sealed box, using a range of equipment set out on a bench. Students had 30 minutes to plan their approach and 30 minutes to complete the task. Two assessors used rubrics, judging that four criteria were considerably difficult, providing for improving instruction. Although criterion-based assessments appear to have high face validity, educational researchers who probe congruence between student and assessor perceptions of tasks are more likely to achieve goals for multiple-strategy assessments.
The immediate effect of moving to the new system elevated the distribution at the lower end of the scaled rubric, recognizing that weaker candidates had developed some competence. Students were helped when we provided materials for planning and implementing projects. Students reported their designs for feedback, evaluated their own designs, and continued their substantive investigations. To examine skills in planning, we used a written sub-test that described a problem, with students planning a solution. We expected a repeat of their project planning exercise. However, the lowest correlations were between the written exam and their projects’ implementation.
We thought these results were because engineering design wasn’t a course requirement. Many years later when Sternberg published his triarchic theory of intelligence we saw another explanation. Sternberg distinguished between three components: 1) meta-components: processes used in planning, monitoring and decision making in task performance; 2) performance components: processes used in task execution; and 3) knowledge-acquisition components used in learning new information. Each component had three properties: duration, difficulty and probability of execution, independent in principle (Sternberg, 1985).
In sum, assessing performance evaluated meta-components or “executive processes”. The two assessment situations required students to use different information processing strategies. Written exercises required different, newer domains of learning and training. To develop this skill, students needed to learn to do it automatically, so that different processes in meta-components were brought into play more quickly. Written papers and projects, while demanding meta-components, are at different levels in an experiential-learning continuum. Some executive processing will always be required for a written paper, but we argue that assessment of task performance and the positive stress it creates are an accurate reflection of everyday activities of the engineering “executive mind”. Rubrics for assessing performance were used with minor modifications until the advanced engineering science course was discontinued nearly twenty years later.

Author

• John Heywood, Trinity College Dublin