Rubric Shopping in the Regimes of Assessment: Navigating the Fuzzy Positivism of Study Abroad Evaluation Tools

• In Event: Transitional and Mobility of Students

Proposal

This study investigates the increasing prominence of intercultural and global skills assessment in study abroad administration and pedagogy, and how it influences the practice of international education and the roles it plays in the administrative spaces of U.S. higher education. We define “rubric shopping” as the way international programs and stakeholders move between different assessment tools and implement them in different ways and toward different ends, for a variety of contrasting reasons. Drawing 8 in-depth interviews with administrators, faculty leaders, and international programs staff, as well as the authors’ experience in international education, we explore the diverse functions served by assessment rubrics in real-world educational contexts.

Applying the work of Shore and Wright (2008; 2017), Doerr (2015; 2017), and Slaughter (2004; 2014), we first discuss the rise in popularity of quantitative tools that purport to measure individuals’ and groups’ intercultural abilities, awareness, or “competence,” in the context of neoliberalization and “audit culture” in higher education. Next, we describe the results of our qualitative research, focusing on (1) the diverse relationships different faculty, staff, and administrators have to assessment rubrics and their implementation; (2) their importance in administrative decision making and accreditation processes; (3) their relationship to neoliberalization concerns in international education and its increasing professionalization; and (4) the role these rubrics can play in promoting intentional program design and pedagogy. We conclude by arguing that these rubrics’ power in administrative spaces comes from the reification of authored concepts like “global citizenship” and “intercultural competence,” along with the scientism of the tools themselves. We define “rubric shopping” as the way international programs and stakeholders move between different assessment tools and implement them in different ways and toward different ends, for a variety of contrasting reasons. This cultivates an ambiguous and “fuzzy positivism that is juxtaposed to and undermined by the ongoing practice of rubric shopping, among other factors. Finally, we encourage study abroad practitioners maintain a degree of critical distance from the field of intercultural skills assessment, while recognizing its utility in advocating for particular outcomes and pedagogical interventions.

Our initial interest in intercultural assessment tools and related administrative culture came from our own experiences in several roles. First, we are both active practitioners of faculty-led study abroad programs and have worked across a range of universities and other educational institutions conducting short and long-term programs since 2007. We are both also scholars of study abroad and have conducted ethnographic research work on others’ study abroad programs (redacted), as well as on best practices for international education (redacted), wherein intercultural assessment often plays a central role (Engle, 2013; Hadis, 2005; Tonkin & Quiroga, 2004) Combined, these previous administrative experiences have informed our approach to this investigation, and we draw on that experience in our discussion and analysis. At the same time, we recognize our combined experience represents a narrow slice of the intercultural assessment field, which includes a range of administrators, assessors, trainers, academics, and tool developers, in addition to students and others who are themselves assessed.

In the full-length paper, we discuss prominent patterns that emerged from these conversations, in dialogue with our historical and scholarly research into the tools and associated rubrics. In identifying the themes and patterns that emerged in our interviews as well as reflections on our own encounters (direct and indirect) with intercultural assessment, several struck us as significant and recurrent: (1) the broad variability of the goals and outcomes different tools elevate; (2) the diverse and important roles they play in administrative spaces of the university; (3) tools’ relationship to the market and the professionalization of ideas that originated in academic spaces; and finally (4) their relationship to program design and pedagogical interventions. The meta-theme that encapsulates many of these results can be characterized as the variability of available rubrics and the ways they can serve different pedagogical or administrative agendas. As we lay out below and return to in our conclusion, many of our collaborators discussed what we came to label rubric shopping, in which they shifted between different intercultural assessment tools to find one they felt fit their programmatic goals, financial constraints, personal values, and even accrediting agency preferences.

These include the experience of Alan, who described assessment tools as handy reference points for high-level administrators and accreditors to use in gauging the impact of study abroad. He brought up a common question he had heard from administrators over the years – “why even bother [to study abroad]?” – noting that his assessment numbers gave them a touchstone that helped reify progress toward specific goals and outcomes, as well as the import of those outcomes. He also mentioned that the IDI specifically can help to bring domestic and international diversity advocates together on a campus, as its implementation is often used to track the effectiveness of on-campus, curricular interventions in addition to international education. “Intercultural competence is a widely accepted value, and it’s grown a lot over the past 15 years,” he said, adding that “the instrument makes it real” and convinces many outside his office of their effectiveness.

Author

• Lauren K Collins, University of Colorado