International Testing of a Performance-Based Assessment in Higher Education

• In Event: Valid Assessment of Learning Outcomes in Higher Education: Cross-National Comparisons

Abstract

International assessments in higher education are especially challenging because differences across countries (e.g., educational systems, SES) increase the complexity of testing (Blömeke, Zlatkin-Troitschanskaia, Kuhn, & Fege, 2013). This becomes even more challenging when using performance-based assessments which are becoming more prominent in assessment programs (Kahl, 2008; Penfield & Lam, 2000).
ANVUR collaborated with CAE to adapt, translate, administer, and score the CLA+, a performance-based assessment of critical-thinking and written-communication skills to Italian university students. The objectives of this study were to see if it was feasible to assess Italian students’ skills, conduct a cross-country comparison of Italians and Americans, and validate the importance of these skills in the labor market.
5,853 students from twelve participating Italian universities completed the Italian CLA+ which included a Performance Task (PT) and a set of 20 selected-response questions (SRQs). The CLA+ was delivered on ANVUR’s platform and the PTs were scored by a team of trained scorers. CAE led trainings between April and June 2013. Italian student performance was compared to 4,380 American graduating seniors.
A translated and adapted version of the CLA+ was used in the study. Student responses are measured on three subscales: analysis and problem solving, writing effectiveness, and writing mechanics. The SRQs measure students’ analysis and problem solving skills on three subscales: critical reading and evaluation, scientific and quantitative reasoning, and critiquing an argument. Students were given 60 minutes for the PT and 30 minutes for the SRQs.

The correlation coefficients for the PT scores and the SRQ score were computed. The correlations between the SRQ and the PT scores were low (r = .27) and this may be due to Italian students’ unfamiliarity with standardized testing formats.
At the institutional level, correlations between subsections are as expected, indicating that institutions with students who have high PTs scores also have students with high SRQ scores and vice versa. The performance of the Italian and American students was compared. Students performed comparably on total CLA+ score. Interestingly, the Italian students outperformed the American students by one standard deviation on the SRQs. After much speculation that the Italian students would be unfamiliar with SRQs, they outperformed the American students by a significant amount, indicating that perhaps this item format is not difficult for the Italian students. Approximately 20 Italian firms are going to evaluate the quality of the students who have taken the CLA+, blinded to their scores, to verify whether having these skills corresponds to the employability and desired competencies.
International assessments are challenging (Blömeke et al., 2013), but this study demonstrated that it is indeed feasible to measure critical thinking and written communication skills using a performance-based assessment. Reliability and validity can best be accomplished through a standardization process and close collaboration is essential. Finally, employers in an international context will provide validity evidence that these higher-order skills are important for hiring decisions.

Authors

• Doris Zahner, Council for Aid to Education

• Fiorella Kostoris