Big Data: How Do We Get It and What Do You Do With It?

• In Event: Big Data: A Big Sell or Oversold in Professions Education

Abstract

In health professions education (HPE), technology enhanced tools for teaching and learning (e.g. online lectures, interactive learning management systems, and smart phones) provide evidence of learners’ interaction with content, faculty, peers, and their environment (Webb & Gibson, 2015; Zeide, 2017). Analyzing Big Data in HPE generates specific learning curves for individuals and provides deeper insights about the patterns of educational performance across institutions (Ellaway, Pusic, Galbraith, & Cameron, 2014; Feldman & Chawla, 2015; Pusic, Boutis, Hatala, & Cook, 2015). The purpose of this paper is to present a framework for the use of Big Data in large scale HPE program evaluation. Big Data may aid evaluators in the important work of understanding the just-in-time impact of curriculum implementation successes or failures in multi-site, or multi-year programs (Durand, Decker, & Kirkman, 2014) and has the potential to be a powerful tool for quality improvement.

Framework
Developmental evaluation (DE), also known as adaptive evaluation, is a flexible evaluation approach in emergent, complex, social innovations (Gamble, 2008; Patton, 1994, 1996; Patton, 2011). The purpose of DE is to inform programmatic decisions concerning the direction of a program, prior to a summative conclusion (Tan & Shulha, 2015). While formative evaluation focuses on improvement, piloting, or program strengthening; DE is focused on “creation, reorganization, and emergence” (Patton, 2011, p. 207).
Because HPE occurs in workplace environments, as well as structured classroom settings, DE is a more suitable approach to evaluation than traditional evaluation models. DE is agile, dynamic, and continuous, matching the underlying principles of Big Data. The complex and uncertain environments in which HPE occurs suggest that DE and Big Data principles should become the foundations of evaluation.


Technology systems capture curricular, admissions, patient encounter, and grade data for analysis (Ellaway, Pusic, Galbraith, & Cameron, 2014). Big Data is extracted through queries from a myriad of educational systems (Figure 1). HPE program data are necessarily messy, which is also a characteristic of Big Data. The value of Big Data for DE is in how much data is collected, the availability (speed) with which it is collected, and how data is aggregated for analysis within the DE framework.


Figure 2 depicts the continuous process of DE. Each triangle represents a time point within the curriculum when data are extracted from the educational data warehouse. Data is analyzed to answer questions of interest. The relationship between each of the triangles may be characterized as dialogic. Stakeholders use the evidence to make decisions, for example, does the curriculum as implemented reflect the curriculum as planned? If it does not, this results in “adaptive moves” (Schein, 2016, p. xiv) or workable responses, to address the concerns that arise.


DE is about understanding innovation and change at a systems-level. Big Data helps to facilitate that understanding through evidence for decision-making during curriculum implementation. The DE framework needs testing and validation; however, curriculum implementation is an example of organizational change, and innovative, adaptive evaluation frameworks are needed to organize and use Big Data.

Authors

• Christina M. Cestone, Drexel University College of Medicine

• Hugh A. Stoddard, Emory University