Measuring Student Learning Outcomes in Higher Education

• In Event: Developing and Evaluating the Psychometric Quality of Next-Generation Student Learning Outcomes Assessments

Abstract

Theoretical Framework and Purpose

The objective of this paper is to introduce an approach to define key competencies for college students and design innovative assessments to measure them. Student learning outcomes (SLO) assessment has been widely used in higher education for accreditation, accountability, and internal improvement purposes (Kuh et al., 2014). Given the various limitations of existing assessments, it’s critical to identify a systematic approach that can be used guide the development of a next-generation assessment.

A broad-based review was conducted of influential theoretical frameworks in the SLO area to help identify shared priorities and definitions. We reviewed frameworks such as the Degree Qualifications Profile, AAC&U’s VALUE rubrics, and Assessment and Teaching of 21st Century Skills. Based on the review, six core areas of SLOs were identified, including Critical Thinking, Written Communication, Quantitative Literacy, Civic Competency and Engagement, Digital Information Literacy, and Intercultural Competence & Diversity (see Figure 1 about the process used to identify these core competencies).

Approach

For each identified competency, we reviewed their definitions in existing frameworks, which helped identify common areas of interest for that particular construct (e.g., critical thinking) across the frameworks. We also reviewed existing assessments in terms of their construct definition, assessment format, psychometric properties, and validity and reliability evidence. This review helped identify relative strengths and weaknesses of each existing assessment and also pointed to directions of improvement when we consider a new assessment.

In our assessment design, we offered a clear definition of the target construct, balanced authenticity of the assessment formats with items’ psychometric quality, employed innovative item types to elicit a wide range of student abilities and skills, considered the multidisciplinary contexts of higher education, and addressed accessibility issues by modifying structural features and item types to expand access for students with disabilities.

To establish the psychometric quality of the items, we conducted a large-scale pilot and collected data from 8,500 students from 41 institutions, including both two- and four-year, and public and private institutions. We examined the dimensionality of the assessments, reliability of the total score and possible subscores, performance differences between freshmen and seniors, aka cross-sectional value-added learning, relations of test scores to other academic success indicators such as GPA, related course grades, etc. Preliminary results provide evidence supporting the use of the assessments to measure college learning and learning gains.

Study Importance

The systematic approach we discussed here provides an example on how to design a next-generation assessment to measure student learning outcomes. Our assessment and research efforts have great potential to benefit institutions that are in the process of designing their own assessments or considering adopting an external measure.

Author

• Ou Lydia Liu, ETS