BackHome
Refresh: OffView Personal Schedule
Paper Summary
Share...

Direct link:

Comparing TIMSS (Trends in International Mathematics and Science Study) Advanced With NAEP Grade 12 Assessments in Mathematics and Science

Mon, May 1, 10:35am to 12:05pm, Henry B. Gonzalez Convention Center, Floor: Meeting Room Level, Room 211

Abstract

Purpose
The purpose of this paper is to describe the results of recent research comparing TIMSS Advanced—which assesses students in their final year of secondary school who have taken or are taking advanced courses in advanced mathematics and physics—with the grade 12 mathematics and science assessments of the National Assessment of Educational Progress (NAEP). While seemingly similar because of the content areas and grade levels assessed, the two assessments are built on different frameworks (see NAGB 2010, 2014; Mullis and Martin 2014), utilize unique sets of assessment questions (or, items), and are each designed for particular purposes and populations, which warrants an understanding of their similarities and differences and what each may contribute to our knowledge base on the performance of U.S. twelfth-graders.
Methods, Data, and Results
This paper uses data from recent research that compared the frameworks and items of the TIMSS Advanced assessments in mathematics and science with the grade 12 NAEP assessments in mathematics and physics, respectively. Results from this study are under review by the sponsoring agency and so cannot be made public until their release on November 29, 2016. This proposal thus describes the type of data proposed for presentation at AERA 2017.
The study’s methodology is adapted from earlier comparison studies (e.g., Gattis et al. 2013). It involved a small group of content experts reviewing the frameworks for each of the assessments and subjects, as well as the items.
• For the framework review, both the content and cognitive dimensions were compared. The content comparisons involved a two-step process in which TIMSS Advanced objectives—the finest level of content detail in the frameworks—were first grouped with one or more NAEP objectives that covered overlapping content. The groupings were then rated on the extent of the coverage, or overlap. The cognitive comparisons involved a qualitative review of the descriptions of this domain in the respective frameworks.
• For the item review, items from TIMSS Advanced and NAEP were reviewed side-by-side to determine if the respective items would hypothetically be found in the counterpart assessment.
The results of the study thus include:
• A quantitatively-based assessment of the similarities and differences in the content to be assessed in TIMSS Advanced and NAEP including, for example: the percentage of TIMSS Advanced objectives that were: fully or mostly covered; partially covered; or not covered by NAEP objectives—i.e., unique to TIMSS Advanced.
• Qualitative data on key differences in how the frameworks are operationalized through their items.

Significance
This paper provides participants with a better understanding of what the TIMSS Advanced assessments in advanced mathematics and physics can tell us about student achievement compared to what we can learn from the NAEP grade 12 mathematics and science assessments. Specifically, it provides insight into the content and topics covered by one or both assessments, as well as how that content is operationalized. Such an understanding is important for interpreting the results from the two assessments and obtaining an accurate picture of student performance at the end of high school.

Authors