Conducting a Problem-Based Learning Meta-Analysis: An Example and Best Practices

• In Event: Research Methodologies for Studying Problem-Based and Project-Based Learning

Abstract

Purpose
While there is a large volume of PBL specific meta-analyses (Dochy et al., 2003; Gijbels et al., 2005; Kalaian et al., 1999; Vernon & Blake, 1993; Authors, 2009 among others), and a much larger set of methods papers, innovations, handbooks and textbooks (Borenstein et al., 2021; Cooper, 2016; Lipsey & Wilson, 2001) for meta-analysis generally there is not much coverage of meta-analytic methods specific to the context of PBL. The purpose of this research is to provide guidance on meta-analysis using examples from the PBL literature as well as highlight needs for future work.

Perspectives
Barrows famously did not align early PBL efforts with a particular educational theory (Barrows, 1986). This effort is aligned post-hoc to Barrows with situated cognition (Brown et al., 1989; Hung, 2002).

Methods
Much of this work is devoted to key sections of a meta-analysis manuscript and important processes used to complete them. Starting with review classification, systematic reviews (Albanese & Mitchell, 1993) as are quantitative meta-analyses, listed above. Meta-synthesis (Strobel & Barneveld, 2009) to review meta-analytic reviews and forms not yet attempted in PBL such as qualitative meta-ethnography, qualitative meta-synthesis, and mixed-methods meta-synthesis (Authors, 2018). Research questions, including examples and the importance of tying each research question to the supporting analyses is discussed and disambiguated from supporting analyses that may lack a question.

Data Sources
The search process with an orientation towards broad inclusion (Steel et al., 2021) and replication of meta-analyses are discussed, as are inclusion/exclusion criteria for individual studies. The importance of providing transparency about the search process, see for example Figure 1, is demonstrated.
Figure 1 – Article counts and critical phases of meta-analytic search.

Effect size calculations (Lipsey & Wilson, 2001) and an online calculator (Wilson, n.d.) are listed and discussed. Care is given to discuss the magnitude of effect size with coverage of PBL debates (Albanese, 2000; Colliver, 2000), the origin of Cohen’s scale (1988), and a more modern context (Kraft, 2020). Coding recommendations are made, specifically aligning the data sought to the research questions and providing full data to readers.

Results
The importance of preliminary analysis is discussed such as publication bias (see Figure 2). Robust variance estimation (Hedges et al., 2010) is presented to quantify the independence of multiple outcomes.
Figure 2 - Sample funnel plot with identification of outlier

From these supporting analyses, techniques for various meta-analysis forms are presented to address research questions for overall contribution, subgroup analyses, and prediction through meta-regression.

Scholarly significance
As a roadmap to modern meta-analytic reviews in the PBL space this work pairs well with other symposium papers. Meta-analysis is a key technique to engage policy makers in positions to make decisions about inclusion of PBL in learning spaces. The coverage of prior work enables scholars to close gaps in the review space, such as seeking mean difference gain scores, correlation based meta-analyses, or techniques for reviewing qualitative work. It also enables contributions that build on the robust review work that precedes them.

Authors

• Andrew Walker, Utah State University

• Heather Leary, Brigham Young University