Making the Case for Methods

• In Event: Communicating Empirical Research in Mathematics Education: Insights From Flagship Journal Peer Review

Abstract

Just as the researcher must justify the significance of research questions and the appropriateness of the theoretical framework, the researcher must also thoroughly describe and justify the selection of methods. Indeed, the research questions and the theoretical framework should drive the choice of methods (and not the reverse). In other words, a sufficiently well-specified set of research questions and theoretical framework establish the parameters within which the most productive methods will be selected and developed. However, reviewers often raise concerns related to methodological choices—concerns that reinforce the need for researchers to pay explicit attention to alignment of the methodology.

To examine reviewers’ concerns about research methods, we applied analogous methods to those described in Paper 1. Table 3 provides a summary of results from the analysis of reviews. One of the most common methodological flaws reported in the reviews was a lack of sufficient justification for the chosen methods. Roughly one third of the reviews for manuscripts that were ultimately rejected included this concern, as did one quarter of the reviews for manuscripts that received a decision of Revise and Resubmit. In contrast, this concern was raised in only about 15% of reviews for manuscripts that were ultimately accepted. The lack of justification is exacerbated when the methods are not described in sufficient detail for readers to understand exactly how the data were gathered or analyzed. In roughly 40% of the reviews for manuscripts that received a Reject or a Revise and Resubmit decision, reviewers called for more detailed descriptions of the methods to better understand the choices that the researchers made or the reasoning behind categories or codes.

One useful heuristic for choosing methods begins with the researchers formulating, as precisely as possible, their set of educated hypotheses about what they will find and the claims they hope to make. We base this recommendation on the premise that research should be guided by educated hypotheses—hypotheses about what one expects to find as possible answers to the research questions based on a foundation of earlier empirical and theoretical work. These educated hypotheses necessarily shape the choice of methods. Once hypotheses are generated, the researchers can work backwards to determine the kinds of data that are needed to address these hypotheses and the methods that would yield these kinds of data.

In addition to employing this heuristic, we argue that many of the methodological issues raised by reviewers can be addressed by utilizing an alternative research pathway—one which involves teacher–researcher partnerships working on solving problems of practice through iterative cycles of innovation in tasks and lessons. Engaging in these cycles means that teacher–researcher partnerships are constantly aligning research questions, hypotheses, and methods. They are testing and refining practical hypotheses about how to improve student learning, drawing from both the wisdom of practice and insights from research. The methods used by a partnership in one cycle must be fully described so the next cycle can build on what the partnership learned and on how they learned it.

Author

• Charles Hohensee, University of Delaware