Findings From the Secondary Test and Instruments Synthesis Group

• In Event: A Synthesis of Mathematics and Statistics Assessments: 2000–2020

Abstract

The purpose of this paper is to present findings from an analysis of quantitative tests and instruments from the Secondary Mathematics Education group within the Validity Evidence for Measurement in Mathematics Education project. Our goals were two-fold, first to identify instruments used in mathematics education research over the last twenty years and to then find the associated validity evidence associated with those instruments.
Our search for instruments has been carried out through several different phases. First, we searched 24 peer-reviewed journal articles from 2000-2020 for grades 7-12 instruments. These journal articles were the top 24 peer reviewed journals in mathematics education according to Williams and Leatham (2017) and Nivens and Otten (2017). This initial phase resulted in 1,747 potential articles with instruments used in mathematics education at the secondary level.
Next, we conducted a title and abstract review to determine if there was an instrument mentioned that met the following criteria: (a) reported an empirical or development study, (b) used in mathematics education, (c) produced scores, (d) measured student attributes, and (e) used in grades seven through twelve. After the title and abstract screening, 432 articles were coded as meeting the inclusion criteria (24.7% of the 1,747). The full texts for each of the 432 articles were reviewed to identify the instrument(s) in each article. This resulted in 285 articles containing one or more instruments that met the inclusion criteria for secondary instruments, from which 504 instruments were identified.
After identifying the instruments, the next phase was to search for the validity evidence for each instrument from any timeframe and any source. Using a detailed search procedure, we searched for validity evidence for each instrument based on citations from the original article, Google Scholar searchers, and Google searches for any non-peer reviewed sources (i.e. white papers, technical reports, conference reports, dissertations, etc.). During this phase, we identified eight additional instruments. In addition to those eight instruments, 112 instruments (n = 73 small scale instruments and n = 39 large scale instruments) were excluded from our list during our validity search. These were excluded for the primary reasons. First, some instruments were determined to be repeats. Second, older versions of instruments were combined and only the newest version was searched for validity evidence (i.e., the seventh edition of the SAT is now out of date). Finally, for some small-scale instruments we had pre- and posttests in our database, but upon further review they had the same items and were scored the same way; these were combined into one instrument.
Finally, of the 400 instruments remaining in our database, validity evidence was identified for 241 instruments (60.25%). We are currently reviewing 357 articles with potential validity evidence and coding them for interpretation and use statements, as well as the five sources of validity evidence and reliability from the American Psychological Association, and National Council on Measurement Education Standards’ ([AERA, APA, & NCME], 2014). This work will highlight the validity evidence available for secondary mathematics instruments.

Authors

• Erin E. Krupa, North Carolina State University
  Presenting Author

• Brianna Bentley, North Carolina State University
  Non-Presenting Author

• Daria Gerasimova, University of Kansas
  Non-Presenting Author

• Deborah M. La Torre, University of California - Los Angeles
  Non-Presenting Author

• David Stokes, North Carolina State University
  Non-Presenting Author

• Emily Toutkoushian, The American Board of Anesthesiology
  Non-Presenting Author