What We Know and Where to Grow: Statistics Education

• In Event: A Synthesis of Mathematics and Statistics Assessments: 2000–2020

Abstract

The focus of this paper is the process and results of the work completed by the Statistics Education Synthesis group within the Validity Evidence for Measurement in Mathematics Education (VM²Ed) project. We performed three initial rounds of work with regard to identifying content to be added to a searchable repository of instruments the project is creating. In the first round, we worked to compile a list of statistics education instruments noted in publications from the year 2000 and beyond. We searched journal databases, proceedings papers from the International Conference on Teaching Statistics (ICOTS), and Google Scholar (to capture dissertations and other conference proceedings papers). In this work, we excluded instruments focused on statistics outside of the statistics education domain (e.g., statistics in biology), as well as instruments where the focus was the interpretation of responses to individual items rather than a score from the entire instrument, to reasonably bound the scope of our work. The focus of the second round was a detailed literature search for those instruments on our list to identify publications citing the instrument. We classified each publication as “having used the instrument” and/or “containing validity evidence.” We also noted which instruments were considered single use—meaning the instrument was created and used exclusively by a single researcher or research team. We are currently in the third round, which encompasses the identification and classification of pieces of validity evidence for each instrument found in the publications identified in the second round using the VM²Ed framework.
Within this paper, we summarize the findings from each round, including a discussion of single use instruments and what types and sources of validity evidence are most prevalent for the statistics education instruments. Overall, a high percentage of statistics education instruments are single use: 76% of the 107 instruments on our list; 77% of 36 student attitude instruments, 71% of 33 student knowledge instruments, and 76% of 12 teacher knowledge and attitudes instruments. Additionally, these instruments tend to have a minimal body of validity evidence. We also highlight instruments with broad bodies of validity evidence with respect to the five sources of validity evidence outlined in the 2014 Standards for Educational and Psychological Testing (AERA et al., 2014). For example, the family of instruments measuring self-efficacy to teach statistics in middle and/or high school, the Self-Efficacy to Teaching Statistics in High School (SETS-HS) (Authors, 2019b) and in Middle School (SETS-MS) (Authors, 2014) instruments (Authors, 2019a), had a total of nine articles that, taken together, addressed four out of the five sources of validity evidence. Our findings suggest that the statistics education field has room to grow in terms of (1) reducing the prevalence of single use instruments, (2) using modern measurement language to frame validity evidence and argumentation, (3) reducing questionable measurement practices (Flake & Fried, 2020), such as combining items from existing instruments into a new instrument without addressing its validity evidence, and (4) treating validation as a program of research (Bandalos, 2018).

Authors

• Leigh M. Harrell-Williams, University of Memphis
  Presenting Author

• Stephanie Casey, Illinois State University
  Non-Presenting Author

• Charlotte Bolch, Midwestern University
  Non-Presenting Author

• Hartono Tjoe, Pennsylvania State University
  Non-Presenting Author

• Douglas Whitaker, University of Florida
  Non-Presenting Author

• Taylor Mule, University of Memphis
  Non-Presenting Author