Methodological Trade-Offs for Research-Practice Partnership Evaluation

• In Event: Assessing the Effectiveness of Research-Practice Partnerships: Lessons From the Field

Abstract

Research-practice partnerships (RPPs) have emerged as a promising strategy for improving connections between research and practice in K-12 education. RPPs are long-term collaborations among educators, researchers, and community organizations that aim to investigate problems of practice in educational systems levels (Coburn, Penuel, & Geil, 2013). Interest in RPPs spans from policymakers (e.g., Conaway, Keesler, & Schwartz, 2015) and advocacy groups (e.g., Data Quality Campaign, the Center for American Progress) to multiple funding agencies, including the William T. Grant Foundation, Spencer Foundation, Hewlett Foundation, and National Science Foundation.

It is not yet clear how well RPPs are achieving their aims. The RPP Effectiveness Framework represents an important beginning point in mapping outcomes on which RPPs could be assessed (Henrick, Cobb, Penuel, Jackson, & Clark, 2017). Yet, the field lacks a deep understanding of where, when, and how we might gather evidence to support claims of RPP progress on their goals. Further, there are trade-offs of different methods for gathering valid and reliable evidence about RPP effectiveness. These issues are further complicated when the resulting data are used for accountability, traditional research, or improvement contexts. Given the recent investments in time, energy, and funding in RPPs (Coburn & Penuel, 2016), these measurement challenges are critical for the future of the RPP field.

We explore these issues in this conceptual paper. First, we present a framework for thinking about RPP effectiveness evaluation, drawing on evidence-centered design tenets (ECD; Mislevy, 2006; Mislevy & Haertel, 2006; Mislevy, Steinberg, & Almond, 1999). Here, in all phases of measure development, a clear sense of each construct guides decisions about what is to be measured; when, where, and with whom measurement will take place; and what methods will be used to gather data. Next, we draw on the efforts from several studies of research-practice partnerships and research use (i.e., Farrell et al., 2018a, 2018b; Farrell, Coburn & Cheng, 2018; Penuel et al., 2016; Penuel et al., 2017) to illustrate four different methods for assessing dimensions of RPP effectiveness: interviews, surveys, observations, and document analysis. Finally, we outline the affordances, constraints, challenges, and key considerations associated with different methods when drawing conclusions about an RPP’s effectiveness on a particular construct. For example, survey items about RPP participants’ progress on particular goals can be administered relatively quickly and cheaply to a broad group of individuals. However, surveys usually depend on retrospective self-reports which can suffer from memory smoothing. It is also difficult to put boundaries around who in the partnership are the best informants to report on RPP progress (i.e., researchers, practice partners, or both) or how many informants are needed to make claims at the partnership level.

This paper will outline the considerations for developing valid and reliable set of instruments for basic research, evaluations, or improvement efforts of RPPs. These issues are central to the development and use of any measures if we are to better understand partnerships’ progress on their goals.

Authors

• Caitlin Farrell, University of Colorado - Boulder

• William R. Penuel, University of Colorado - Boulder