Self-Direction: Building Assessment Validity Through Practice and Research Dialogue Within a Continuous Improvement Framework

• In Event: Fostering Deeper Learning in Research-Practice Partnerships: Theoretical and Practical Perspectives

Abstract

Overview
New Hampshire’s Performance Assessment for Competency Education (PACE) system built a valid and reliable alternative for measuring content mastery comparable to state-level standardized testing. Assessment of legislated Work-Study Practices (aka deeper learning competencies) were next. The overall study sought to understand the diffusion and scaling of Work-Study Practices through PACE, invitational teacher professional development, and relational dynamics between research and practice domains. This poster is a phenomenological exploration of one scaling mechanism – development of assessment validity for self-direction.
Theoretical Framework
Assessments in educational settings must ensure they measure what they say they are measuring. In other words, their interpretation and use must ensure validity (Miller, Linn and Gronlund, 2012: 71). Performance assessments pose unique challenges in constructing validity arguments (Lai, Wei, & Fulkerson, 2012) demanding evidence of consensus on the meaning and interpretation of constructs being measured. Research practice partnerships (RPPs) bring together key strands of expertise that can built towards “know-how” (Nelson et al, 2011). Dialogue works to weave varied expertise together helping to move towards shared meaning and understanding (Feng & Chen, 2011; Phillips, 1994). Continuous improvement complements dialogic processes by focusing strands of expertise on shared problem-solving, solution-finding, and joint learning (Cohen-Vogel et al, 2015; LinkedIn, 2017).
Methods and Data Sources
The study is a phenomenological exploration of one scaling mechanism – the creation of validity in the performance assessment of self-direction. The study involves deep participant observation of an RPP in New Hampshire. The RPP included three dozen classroom teachers from ten school administration units, a dozen teacher and administrative leads from four pilot study districts, staff from a statewide education intermediary working with practice and research consultants, the state educational agency, and a national nonprofit intermediary in the education and workforce space. Additional data collection included review of practice artifacts from spring 2018 to fall 2020, interviews with key stakeholders, teacher surveys, student focus groups, and student level performance data within a grounded theory analytical frame.
Results
The exploration revealed continuous improvement cycles driven by dialogue between researchers and practitioners worked to address both face and construct validity. Iterative cycles of design, testing, feedback and analysis involved inputs from students, classroom teachers, teacher leaders, professional development experts, educational research experts and an established research framework related to deeper learning skills. These strands of expertise built and created shared meaning and confidence in the various assessment dimension of self-direction through the co-creation of a field facing rubric and evidence collection tools. The resulting tools and rubric increased both the face validity informed by practice expertise and construct validity through research expertise and alignment with an existing research framework.
Scholarly Significance
The study provides insights into the benefits of co-creation of performance assessment for competency education by both practice and research expertise. It highlights how practice expertise deepens the research enterprise and vice versa. It also demonstrates the need to include not only educators but students in the process of performance assessment design as key to validity.

Author

• Felicia M. Sullivan, Jobs for the Future