A Collaborative Approach to School Improvement Efforts: Partnering with Youth to Validate Outcome Measures

• In Event: The Successes and Challenges of Measurement in School and Out-of-School Improvement Research

Abstract

A key component of Improvement Science is using participatory approaches to engage stakeholders closest to the problem in evaluative activities. Advocates of participatory methods suggest empowering youth as co-researchers generates more reliable data, drives richer data exploration, facilitates deeper data insights, and empowers data-driven decisions (Ozer & Douglas, 2013; Powers & Tiffany, 2006). To date, active participation of youth has been leveraged at various stages of the research process, including recruitment (e.g., respondent-driven sampling), data collection/analysis (e.g., peer interviews), and program implementation (e.g., delivering program content). Yet, despite its lauded benefits, the use of participatory approaches to improve measurement validity is an area that remains largely unexplored (Gonzalez & Trickett, 2014).

The current study describes a research-practice partnership between a racially, ethnically and socioeconomically diverse group of 36 youth researchers (grades 7-12) and a university evaluation team. Together, youth and adult partners engaged in an iterative cycle of problem identification, solution design, and rigorous testing aimed at driving the continuous improvement of the Compass model. Developed by Valor Collegiate Academies and adopted by 64 partner schools across the country, the Compass model is a school-based framework for comprehensive human development emphasizing the central role of relationships in everything from academic curriculum to classroom practices, and from exploration of individual identity to the collective establishment of school culture. A key outcome measure of interest in the improvement efforts is a sense of school belonging. Broadly defined as the extent to which students feel accepted, respected, included, and supported, sense of school belonging has been open to numerous definitions, conceptualizations, and methodological approaches (Goodenow & Grady, 1993). However, the underlying assumption remains that it is a socially contingent, culturally-anchored construct that changes over time. Accordingly, a participatory approach was applied to examine the content validity of a frequently used measure: the Psychological Sense of School Membership Scale (PSSM; Goodenow, C., 1993). The PSSM includes 18 statements rated on a 5-point Likert-type response scale (e.g., “I feel very different from most other students here”; see Figure 1). Using the Key Informant Validity Index (KIVI; Author, 2022), youth researchers conducted a two-stage mixed method validation process to determine whether survey items maintained adequate levels of content validity when applied to the local population (i.e., a study sample that differed from the original norming and validation study samples). Each item on the survey was rated for relevance, clarity, and coverage, and a guided protocol was then used to conduct cognitive interviews further exploring their peers’ interpretation and responses to survey items. Four statements were flagged as problematic (i.e., proportion of agreement at the item- level did not meet threshold; see Table 1). Recommendations were made to remove two items (due to low relevance), revise the wording of two items (to increase clarity) and to add two items aligned with emergent themes from interview responses.

This exploration of participatory approaches opens the door to directly engage with local stakeholders and generate knowledge in partnership with youth to advance measurement validation procedures.

Author

• Shereen El Mallah, University of Virginia