Mixed-Methods Approaches to Evaluation in an Educational Setting for Effective and Holistic Storytelling

• In Event: Truth, Its Beauty, and Its Justice: Reconceptualizing How Stories Are Told in Educational Evaluation

Abstract

This paper presents a cumulative summary of a research and evaluation project of the Incubating Engineers for Food Innovation (iNEST) Program which is a collaborative project among several institutions of higher education including the University of Tennessee at Chattanooga, University of Tennessee at Knoxville, and Tennessee Technological University, between 2021 and 2024. Based on some unforeseen programmatic challenges, including those presented by the COVID-19 pandemic, the evaluation approaches had to be modified to enable the researchers to tell a holistic story regarding different aspects of the program in a way which included the voices of every stakeholder, including students and faculty. In this paper, lessons learned from the research and evaluation will be discussed, including how we used triangulation, to achieve the most from the process and tell an inclusive and holistic story.

Different assessment approaches were utilized in order to provide a more complete picture of the whole project, enabling the “story of the iNEST program to be told,” and for the program to be an example for future researchers and implementers of similar programs. We used a mixed methods approach which included snapshot assessments (surveys) following Professional Development and Effective Learning workshops, interviews with course instructors and the PI team, focus groups, and document analysis. All assessments were conducted virtually.

One of the key reasons for the success of the evaluation and of the project overall emanated from the transparent communication between the researchers/evaluators and the PI team, with buy-in from all stakeholders to implement formative assessment feedback. During planning and reflection meetings with the faculty and staff, the evaluators provided actionable items for each relevant stakeholder category. In this way, stakeholders’ voices were able to be heard. Secondly, the use of retrospective pretest surveys to assess (perceptions of) knowledge and skills gained is one of the key pieces of the evaluation approach, which has been very informative in terms of shedding light on the teaching and learning of food science and technology across diverse students.

We believe sharing our experience evaluating this (still ongoing) program will help other evaluators in education and research to further reflect on their approaches to evaluation in order to be able to tell their stories more effectively while also examining how their own lived experiences impact the work that they do.

Authors

• George Chitiyo, Tennessee Technological University

• Kinsey Simone, Tennessee Technological University

• Michael Danquah, University of Tennessee

• Marlana R. Lastres, Tennessee Tech University