Answering the "So What?" Question

• In Event: Communicating Empirical Research in Mathematics Education: Insights From Flagship Journal Peer Review

Abstract

A frequent question posed to authors by reviewers is “So what?” In other words, reviewers are asking, “What difference do your findings make, and how do your results advance the field?” All researchers wrestle with articulating a convincing argument about the importance of their work. In making this argument, it is easy to fall into the trap of making claims about the implications of our findings that reach beyond the data.

To examine reviewers’ concerns about interpretation of the findings, we applied analogous methods to those described in Paper 1. Table 4 provides a summary of results from the analysis of reviews. We found that reviewers often raised concerns about authors’ interpretations, including about claims in their report that did not match the data presented in one of two ways. On the one hand, reviewers raised concerns about claims which had insufficient support but which reviewers felt a more extensive or careful analysis of the data collected could potentially support. For manuscripts that received a decision of Accept with Revisions, the majority of concerns about support for claims fell into this category. On the other hand, reviewers raised concerns about claims they felt the data and analysis were incapable of supporting. These issues are severe because they break the chain of reasoning that aligns the research questions, the theoretical framework, the methods, and the findings. For manuscripts that were rejected, the majority of reviewers’ concerns about support for claims fell into this category.

Reviewers also noted when manuscripts needed stronger arguments for the significance of the findings. About one third of the reviews for rejected manuscripts included such concerns. Even for manuscripts that were accepted, 14% of reviews included some concerns about the significance of the work. However, in many of these cases, reviewers offered suggestions to the authors about how to strengthen their argument. A key ingredient to the significance of the study is the significance of the research questions. Specifically, the importance of the findings, and of the study itself, emerges from justifying the significance of the research questions. Interpreting the findings then becomes the final link in a chain of reasoning that extends back to the theoretical framework and the research questions.

To summarize, all four papers presented here emphasize justification, coherence, and significance. Justification is necessary at every step of research: arguing for the significance of research questions, making clear why a methodological choice is appropriate, and convincing readers that interpretations of the data are well supported. Coherence requires that all of the components of research fit together into a consistent narrative—a chain of reasoning that connects the theoretical framework, research questions, methods, and ways in which results are presented and interpreted. Finally, especially for academic publication, research must be significant, advancing our knowledge and understanding of teaching and learning substantively. This analysis of reviews for a flagship journal in mathematics education provides insights into how reviewers are likely to evaluate manuscripts with respect to these ideas.

Authors

• James Hiebert, University of Delaware

• Michelle Cirillo, University of Delaware