Dilemmas in Assessment Development for Youth Civic Learning

• In Event: Telling Fact From Fiction and Other Challenges in Enacting High-Quality Civic Education

Abstract

Objectives

This paper shares findings from the validation process for the “Measure of Youth Policy Arguments in Authentic Contexts” (MYPA). MYPA is an observation protocol, meant for formative and summative uses, that assesses the quality of student culminating performances in action-civics projects (Levinson, 2012). Action civics engage students in cycles of problem-identification, research, analysis, and public policy presentations tied to topics relevant to their lives.

Although action civics is gaining increased support from educators and scholars, few educational resources exist to support youth in constructing and delivering high quality, evidence-based policy arguments to authentic audiences (Pope, Stolte, & Cohen, 2012). Such presentations are often the first time that young people step into the public sphere and speak to public officials; they represent rich opportunities for academic skill development and identity formation (Author, 2010; Su, 2013).

Perspectives

We draw on a situated and critical approach to assessment validity that emphasizes the importance of context, relevance to education practice, and opportunities to learn (Gee, 2010). Our view is also influenced by an argument approach to validity that emphasizes the interpretation and uses of scores (Bell et al., 2012; Kane, 2008; Maul, in press). These perspectives inform our discussion of dilemmas related to our intended goals for MYPA, who will use it, and in what kinds of educational settings.

Methods & Data Sources

We developed research-practice partnership (RPPs) with two community organizations and one school district to focus on developing, piloting, and evaluating the usefulness of MYPA. Data sources include field notes, focus groups with policy and measurement experts, video footage of youth presentations, and surveys documenting feedback from teachers.

Results

Dilemmas about whether to prioritize research practice partnerships or psychometric precision. Because of the open-ended nature of culminating performances, which can be about any topic and delivered in a range of settings, scoring video-taped presentations required ongoing refinement of decision-rules and item language. But if we devoted all of our time to demonstrating internal reliability, we would have less time to develop partnerships with educators to support action civics and gather data about the protocol’s usefulness.

Tensions between epistemological assumptions about measurement and normative conceptions of democratic citizenship. Measure development often assumes that item descriptions capture objective distinctions of quality along a single dimension. This assumption is in conflict with normative and value-laden qualities of democratic action and views of social justice. This tension showed up most visibly in discussions about how to measure problem framing and use of evidence.

Tensions between academic standards for communication and political norms for communication. Academic writing and speech typically favors arguments that are dispassionate and weigh evidence; this reflects deliberative norms that emphasize openness to ideas and rational persuasion (Sprain & Ivancic, 2017). Authentic speech in public decision-making settings, however, tends to be more persuasive and adversarial.

Significance

Our paper contributes to theoretical debates about assessment validity and argues for developing assessments in partnership with teachers. We also share a practical educational resource to support student voice and civic development.

Authors

• Ben Kirshner, University of Colorado - Boulder

• Shelley Zion, Rowan University

• Daniela Kruel DiGiacomo, University of California - Riverside

• Ginnie Logan, University of Colorado - Boulder