System and Site Validity: New Frames for Talking About Assessment Change as Cultural Disruption

• In Event: What Do We Mean When We Address the Concept of "Assessment Cultures"?

Abstract

Objectives:
This presentation explores notions of ‘system’ and ‘site validity’ as new frames for ‘reseeing’ assessment change in context. The case instance is assessment in teacher education, and specifically, the introduction of mandatory, high-stakes, summative assessment of graduate competence. The opening that the work provided for digital disruption is also addressed.

Perspective:
The policy driven reform initiative into teaching performance assessments (TPAs) has major implications for teacher educators’ understandings about professional competence and guild knowledge (Sadler, 1989). Noting US experiences with the introduction of TPAs we took up the challenge to work at scale, bringing together universities to collaborate across states, territories and policy contexts to move the assessment narrative from compliance to collaborative professionalism. This saw an intensification of energy around local or site contexts of universities and their respective communities. We drew on perspectives to see assessment as: socio-cultural practice (Broadfoot, 1996); enacted through talk, text and interactions among members of various communities; all knowledge is personal (Polanyi, 1962), involving some grafting of understandings about new policy or system requirements into existing understandings. With contributions from interdisciplinary teams, policy personnel, curriculum specialists, assessment philosophers, and systems thinkers, we brought a perspective of assessment as enabling. This opened a radically different research space where we learnt new ways of thinking, talking, being, sharing, and acting.

Methods:
We conducted first a policy analysis of global, national, state, and local education policies informing assessment culture of teacher education. A second step was to identify conceptions of assessment and teaching competence, held by different stakeholders through applying an ethnomethodological approach to analyzing talk and interactions (e.g., Silverman, 2006; Freebody, 2003). These views are often tacit and reified as ways of doing assessment and so are difficult to articulate, change, and negotiate. Doing so, we interrogated various university’s assessment cultures and underpinning epistemologies of learning that inform teaching and assessment practices valued in that culture.

Results:
The study has generated theoretical knowledge about the nature of change in a significant national assessment policy initiative in Australia. This includes contributions of democratic forces in the lived experiences of those participating in the change across a range of contexts. Voices of teacher educators, preservice teachers, and policy personnel are incorporated in a multivocal commentary on what it takes to get TPAs right in the best interests of learners. The presentation shares insights into digital innovation that was unexpected at the beginning though it has become core to the project.

Significance:
The presentation shows the ecology of change that the project brought forth. We reveal how standards have a utility in making clear expectations of quality, though they can appear as blunt compliance tools rather than promoting problem-solving and collaborative inquiry across different communities. In the project, we ask: What does productive assessment change look like when issues of evidence, agency, and new ways of thinking at system and local levels are placed centre stage? A main outcome is a set of conditions that have potential to sustain productive engagement in critical inquiry into assessment reform.

Authors

• Claire M. Wyatt-Smith, Australian Catholic University

• Lenore Adie, Australian Catholic University

• Chantelle Day, Australian Catholic University