On the Practices and Challenges of Measuring Higher Education Value Added: The Case of Colombia

• In Event: Valid Assessment of Learning Outcomes in Higher Education: Cross-National Comparisons

Abstract

Policy makers are increasingly asking colleges and universities to assess student learning outcomes, and in some cases to provide evidence of the value they add to student learning. There is widespread recognition that the “quality” of student intake into higher education institutions as measured by, for example, test scores, varies tremendously and that higher education outcomes vary correspondingly. Hence, policy makers and educators have voiced concerns about “fairness” or have called for “leveling the playing field” by adjusting learning outcomes for intake quality.
The concept of “Value Added” (VA, e.g., Kim & Lalancette, 2013) is an attempt to do so. The gist of the VA methodology is as follows: Based on the characteristics of students admitted to an institution we can predict those students’ achievement test scores at the end of college. The VA of studying at college for a student is the student’s observed score on an outcome measure minus the student’s expected outcome based on a prediction from their college-entry characteristics. All other things equal, two students with exactly the same entry characteristics attending different colleges are considered to achieve differently because one college added more value to a student’s achievement than another.
In this paper, we explore the difficulties of using such an approach by describing the challenges we have faced when attempting to implement such a system in Colombia. The purpose is to examine the challenges and practices of measuring VA in higher education by describing the requirements and assumptions underlying its estimation (manipulability assumption, table-unit treatment value assumption (SUTVA), interval scale assumption, homogeneity assumption, strongly ignorable treatment assumption, functional form assumption) as well as the practical decisions which underlie VA measurement (specification of the treatment, choice of the unit of analysis, decisions about outcomes). Using data from Colombia, we will highlight cases where these assumptions are challenged by the nature of higher education and come into conflict with the practical realities. Colombia has an assessment system suitable for the production of VA estimates for virtually the entire system of colleges in the nation. All high school and college leavers take common, national examinations (SABER 11, SABER PRO), which provide us with longitudinal data, which was used for the analyses in this study. We will give examples of VA estimates and will show that VA models and estimates can be considered very delicate instruments. Finally, we will draw implications for policy makers, educators and the public.

Authors

• Richard J. Shavelson, SK Partners & Stanford University

• Benjamin W. Domingue, University of Colorado - Boulder

• Julian Patricio Mariño von Hildebrand, Universidad de los Andes

• Adriana Molina, Colombian Institute for Educational Assessment

• Julián P. Marino

• Edward Wiley