"It Has to Be the Common Thread": Weaving Attention to Equity Across Evaluative Criteria

• In Event: Toward Racial Equity and Justice in Educational Evaluation: Research on Conceptualizations, Criteria, and Evaluator Practices

Abstract

1.Purpose
Evaluators must attend to equity in each step of the evaluation process (Hood, Hopson, & Kirkhart, 2015; Thomas & Campbell, 2021). This paper takes up the step(s) in which evaluators determine the study’s focus by formulating evaluation questions, specifying indicators, and identifying needed data. These lines of inquiry are shaped by implicit or explicit evaluative criteria that represent values about the characteristics that define a “quality” program (Davidson, 2005; Patton, 2021). Little is currently known about how evaluators specify and investigate equity-focused lines of inquiry (AUTHOR, YYYYc; Gates et al., 2022). Therefore, we posed three questions: (1) What lines of inquiry do evaluators pursue when assessing initiatives that aim to foster racial equity? How do they define and investigate quality? (2) Whose values shape those definitions of quality? (3) What challenges do evaluators encounter?

2.Theoretical framework
We used an integrated model of evaluative criteria domains and sources (AUTHOR, YYYYb, YYYYd) as our conceptual framework. Criteria domains describe the overarching focus of a line of inquiry (Table 1), while sources of criteria describe the people or documents that specify or prioritize that focus (Table 2).

3.Methods
Previous studies employed surveys or analyzed evaluation reports to investigate evaluative criteria (e.g., AUTHOR, YYYYd; Shadish & Epstein, 1987). However, those methods capture limited detail and context, pointing to the need for interviews with evaluators (AUTHOR, YYYYb, YYYYd). We conducted semi-structured interviews (Patton, 2015) informed by an interpretive, qualitative methodology (Glesne, 2016). In this paper, we report analysis of interviews with 21 educational evaluators with experience conducting equity-focused evaluations with BIPOC communities. We analyzed transcripts using a priori codes (Miles, et al., 2019) representing the criteria domains and sources in our conceptual framework. Inductive coding (Miles et al., 2019) captured (a) additional domains and sources and (b) challenges evaluators encountered.

4.Findings
Emerging findings reveal multiple equity-focused lines of inquiry evaluators pursued, through formal data collection and informal assessment and feedback. Evaluators investigated transformative outcomes, outcome disparities, program climate and reach, whether program designs addressed structural inequities, and programs’ relevance to community circumstances. This inquiry reflected criteria drawn from evaluation commissioners and program leaders, as well as evaluators’ frameworks and community priorities. Rather than investigating equity as a stand-alone dimension of program quality, evaluators examined equity across criteria domains. As one respondent explained, attention to equity “has to be the common thread” woven across lines of inquiry. Evaluators encountered challenges reaching a clear understanding of what equity “looks like”, determining the appropriate scope of change to investigate, and addressing unrealistic or inappropriate target outcomes.

5.Significance
The literature on evaluative criteria describes equity-focused inquiry as examining outcome disparities (Rogers, 2016) or programs’ cultural relevance (Schwandt, 2015) and calls for research to understand how evaluators address equity as an evaluative criterion (AUTHOR, YYYYc; Gates et al., 2022). Our findings reveal multiple equity-focused lines of inquiry investigating varied dimensions of program quality. This illuminates how equity can be examined across criteria domains, functioning as a cross-cutting lens rather than a stand-alone domain.

Authors

• Rebecca M. Teasdale, University of Illinois at Chicago
  Non-Presenting Author

• Cherie Avent, University of Illinois at Urbana-Champaign
  Non-Presenting Author

• Ceily Moore, University of Illinois at Chicago
  Presenting Author

• Xinru Yan, University of Illinois at Urbana-Champaign
  Non-Presenting Author

• María B. Serrano-Abreu, University of Illinois at Urbana-Champaign
  Non-Presenting Author