Findings From the Instruments for Elementary Students Synthesis Group

• In Event: A Synthesis of Mathematics and Statistics Assessments: 2000–2020

Abstract

This study presents findings from an analysis of quantitative instruments related to elementary school children from the last 20 years of the mathematics education literature. This analysis is part of a larger project designed to develop a repository of instruments and their associated validity evidence for use. We describe the frequency with which different types of elementary tests appear in the mathematics education literature, as well as determining the extent to which elementary tests are constructed, reported, or taken up in ways that represent intent for use by the community of mathematics education researchers.

Methods
The research team started with an initial list of journals related to elementary (K-6) mathematics education based off the lists found in Williams and Leatham (2017) and Nivens and Otten (2017). We identified a list of search terms to use within these specific peer-reviewed journals and limited the search to 2000-2020. Based on this initial search criteria, we developed a list of articles that included elementary mathematics instruments. We identified the instruments used within these articles and searched for validity evidence within that article or related articles. When identifying instruments, we categorized types of instruments of intended use (single-use, multiple-studies or broad use). We also identified whether there was an interpretation or use argument and the types of reliability and validity evidence provided. Throughout this process, the research team met quarterly to monitor progress and calibrate. We checked for the consistency of our coding by double-coding a sample of 10% of the articles. Discrepancies were discussed collectively, and revisions were made based on the discussion.

Results
A total of 192 distinct measures of elementary students were observed across the sample of journal articles. Approximately half of the measures (51%) identified were classified as being used only once. These single-use measures were specific to the particular study and not designed for use by other researchers or used across multiple studies by the same researchers. Measures designed for broad use accounted for 21% and measures used across multiple studies but not necessarily broad use accounted for 28%.
Overall, there was a low percentage of instruments with explicit interpretation statements and very few instruments with validity arguments to support the use of the measure for intended purposes. We found a statistically significant difference in prevalence of interpretation statements (H(df=2) = 25.38, p < .001), with single-study use having statistically fewer incorporations than broad (p < .001) or multi-study use (p < .001) measures. We also found a statistically significant difference in prevalence of use statements (H(df=2) = 29.74, p < .001), with single-study measures having statistically fewer incorporations than broad (p < .001) or multi-study use (p < .001) measures. Additionally, multi-study measures had statistically fewer incorporations than broad use measures (p = .03).

References

Nivens, R. A., & Otten, S. (2017). Assessing journal quality in mathematics education. Journal for Research in Mathematics Education, 48(4), 348–368.
Williams, S. R., & Leatham, K. R. (2017). Journal quality in mathematics education. Journal for Research in Mathematics Education, 48(4), 369-396.

Authors

• Jeffrey C. Shih, University of Nevada - Las Vegas
  Presenting Author

• Marsha M. Ing, University of California - Riverside
  Non-Presenting Author

• Cindy Jong, University of Kentucky
  Non-Presenting Author

• Karl W. Kosko, Kent State University
  Non-Presenting Author