Misconceptions About Data-Based Decision Making in Education: An Exploration of the Literature

• In Event: The Complexity of Data-Based Decision Making: Evidence From the United States, Norway, New Zealand, and Belgium

Abstract

Purpose
Research on DBDM has proliferated around the world, fueled by policy recommendations and the diverse data that are now available to educators to inform their practice. Yet, many misconceptions and concerns have been raised by researchers and practitioners. This commentary is an exploration of the criticisms grounded in research, theoretical, and practical evidence to dispel some of the misconceptions. The objective is to survey and synthesize the landscape of the DBDM literature to serve as a stimulus to changes in policy and practice as well as a roadmap for a research agenda.

Theoretical framework
The paper, which was the lead article in the special journal issue, examines a broad range of literature in the field of data use. It was guided by existing theories of data as a process of collaborative inquiry. The paper takes an objective position around key topics found in the field of data against which criticisms have been leveled, then identifies how the existing literature addresses each of those criticisms.

Method and data sources
The paper used a literature review in addition to the authors’ extensive knowledge of the data field to identify key studies to be cited to either refute, confirm, or further discussion around each criticism. Data sources consisted of the existing literature for data use.

Results and conclusions
The paper addresses the supporting or disconfirming literature around the following misconceptions and issues: (a) that DBDM making lacks a theory of action; (b) that the purpose of data use is for accountability and compliance rather than continuous improvement; (c) that data are only test results, thereby taking a narrow view of what are educational data; (d) relatedly, the conflation between assessment literacy and data literacy prompting the need to consider data literacy as a broader and more inclusive construct; and (e) that some data technologies misrepresent data by “dumbing down” the representation to a stoplight – that is, green for passing, yellow for cautionary, and red for failing.

Scientific significance
This review provides a credible and objective discussion around the criticisms that continue to plague the field of data use. It provides a balanced discussion on several misconceptions and also outlines how future research can address the gaps in knowledge and advance the field through theoretical, methodological, and practical foci in new avenues of research. The paper outlines specific recommendations around each topical area where research is needed, new methodological considerations, and what the impacts might results for practice and policy in terms of educational data use.

Authors

• Ellen B. Mandinach, WestEd

• Kim Schildkamp, Universiteit Twente