Illustrating a Robust Evaluation Framework for Sustainability Focused Leadership Education Programs

• In Event: Symposium | Enabling Non-Leadership Faculty to Teach Leadership Concepts to Address Complex Contextual Problems

Short Description

Leadership education programs can embrace heliocentric evaluation frameworks that utilize mixed methods to collect a robust and diverse set of evidence to tell the story of program impact. Outcomes-oriented evaluation practices will be discussed and illustrated using a real-life leadership education example and accompanying data. This presentation will show participants potential evaluation strategies which can be used to (a) identify measures of acceptable evidence; (b) articulate short, medium, and long-term outcome indicators based on established outcomes; and (c) plan appropriate educational activities aligned with them.

Detailed Abstract

Conceptualizing emerging and established evaluation strategies is a critical component for consideration as we collectively reimagine leadership. Evaluation strategies that have stood the test of time include the Kirkpatrick model (1994), CIPP model (Stufflebeam & Corny, 2014), Logic model (LeCroy, 2018), and Goal-Free model (Scriven, 1991). The essence of a robust leadership education program evaluation is that it should be an unusually systemic and persistent process for understanding the value of a given endeavor. Scriven (2010) notes that historically, evaluation has been built and revolved around a given activity, referring to this as geocentric focus. Scriven laments this focus and calls for a more heliocentric approach whereby evaluation is the central focus around which activities are built. This approach is consistent with Locke’s (1968) goal setting theory and Wiggins and McTighe’s (1998) backward design approach that begins with outcomes in mind. Our project team used an evaluation framework based on well-established evaluation strategies that embraces Scriven’s (2010) heliocentric approach.

To illustrate this type of approach, consider the case of the grant project entitled “Preparing Organizational Leaders in Agriculture through Innovative Case Studies Contextualized in Agricultural Disasters” (POLA). The POLA project was designed to teach non-leadership faculty how to integrate the use of leadership concepts into their bench science undergraduate courses by engaging participants in an online faculty development academy followed by a capstone field experience. Intended outcomes were first described as (a) increased knowledge of Bolman and Deal’s (2017) leadership frames, (b) increased motivation to embed leadership concepts into their courses, and (c) increased ability to develop case studies integrating leadership frames with technical content. With identified outcomes, the POLA team created a robust leadership education program built around an evaluation framework based on a pragmatist approach that gathered a mix of quantitative and qualitative data (Ary et al., 2010; Creswell & Plano Clark, 2006).

Quantitative methods to measure knowledge gain often involve the use of t-tests. For outcome (a) of the POLA project, knowledge of leadership frames was measured using a pre- and then post-test design (Rossi et al., 2004). Evaluators should collaborate with content experts when using this approach to improve the content validity of the test; the POLA project employed three leadership experts for this purpose. The comparison of pre- and post-test scores provided initial evidence for tracking the leadership competency development of participants. Knowledge gain is generally regarded as an essential precursor to practice change (Taylor Powell et al., 1996).

Another set of data was obtained by utilizing qualitative methods. Guba and Lincoln (1981) indicated qualitative research is a credible approach for evaluation researchers to understand programmatic impact through an analysis of participants’ verbal or written voice. A qualitative approach was used for outcome (b) to explore faculty perceptions of the online faculty development academy and field experiences towards enhancing their knowledge and application of Bolman and Deal’s (2017) leadership frames to develop case studies. Data collection methods from the POLA project field experiences included focus groups (Krueger, 2009), in-depth semi-structured individual interviews (Maynard et al., 2016), participant observation (Patton, 2014), and records (Hatry, 2004). Reflective guides, interview guides, and observation schedules were developed by the POLA project’s team. Lincoln and Guba’s (1985) methods for establishing trustworthiness, including member checks, audit trails, and triangulation of data were followed to ensure rigor. The qualitative methods assisted in determining the extent to which the project influenced participants’ leadership education teaching abilities, and in turn, the quality of instruction to prepare students for leadership roles post-graduation.

Finally, an additional qualitative method was used for outcome (c) to evaluate whether participants had learned the leadership content well enough to be able to teach it to a secondary audience (their students). Evaluating this ability was accomplished by creating an online survey instrument that included an open-ended item which asked students to describe what they learned from the case study. An example of the responses received through this method was: “I learned that the four leadership frames can be used in various ecosystems, especially those in Florida, to improve sustainable land and water management practices and help rebuild when natural catastrophes occur.” An analysis of the frequency of student responses that mentioned leadership generally, or the leadership frames specifically, provided insight as to student awareness. Further, the responses provided by the students were analyzed for accuracy to assess student knowledge. Taken together, these outcome indicators provided evidence of program value based on the central evaluation framework; this may not have been possible if the methods had only focused post-reflection from the participants and activities.

As a case study, the POLA project demonstrates the effectiveness of the heliocentric approach to program evaluation in leadership education. The project utilized a pragmatist epistemology, created a tailored evaluation model built on enduring evaluation frameworks, and developed program activities that complemented pre-identified evaluation outcomes. As a result, evaluation data provided sufficient evidence of program value consistent with the overarching aim of the POLA project. The heliocentric approach is well-suited for evaluating educational programs since it provides a rationale for decisions on specific program activities. Similarly, major program development models typically identify gaps or problems before describing program components such as target participants, activities, and inputs. These gaps directly translate to desired program outcomes, analogous to performance or behaviors in the goal setting theory and desired results in the backward design approach. Evaluators can create performance indicators, measures of acceptable evidence, or short, medium, and long-term outcome indicators based on established outcomes identified through a heliocentric approach. Following, a program team can select and/or design program activities that target outcome indicators. The heliocentric approach to program evaluation enables a logical sequence of actions that eventually leads to educational activities that are consistent with leadership program goals.

Participants

• Amy Harder, University of Florida
  Presenter

• Lendel Narine, Utah State University
  Presenter