Diagnostic Studies Using TIMSS and TIMSS Advanced Assessment Data

Group Submission Type: Formal Panel Session

Proposal

Introduction
For the past few decades, the focus on science, technology, engineering, and mathematics (STEM) education has grown, with many countries seeking to increase the number of students who pursue further study and careers in STEM and to ensure STEM literate citizens. It is thus important to identify which science and mathematics concepts are problematic for students to determine what changes may be needed in the school curricula and instruction to improve the teaching and learning of these key subjects throughout elementary, middle, and secondary school. The Trends in International Mathematics and Science Study (TIMSS) is a flagship study of the International Association for the Evaluation of Educational Achievement (IEA). Participation in TIMSS assessments at grades 4 and 8 and in TIMSS Advanced, which measures the achievement of students in their final year of secondary school who have taken advanced coursework in mathematics and physics, enables countries to evaluate their student performance in science and mathematics across grades in an international context. The main reports released from TIMSS and TIMSS Advanced focus on average scale scores and the percentage of students reaching international benchmarks. Secondary diagnostic analyses of TIMSS and TIMSS Advanced data are needed to further explore student performance and to provide more in-depth results that can be used to inform curriculum and instruction across grade levels.

In this symposium, we present three papers that use TIMSS and TIMSS Advanced assessment items (questions from the assessments) and student performance data from assessments conducted across 20 years (1995 to 2015) to explore students’ levels of understanding of select concepts in physics and mathematics. These papers use diagnostic methodologies focused on specific topics to identify areas of relative strength and weakness and show how performance data on individual TIMSS and TIMSS Advanced items can be used to explore students’ misconceptions and errors across grade levels.

The use of TIMSS and TIMSS Advanced data to explore students’ level of understanding of key topics in science and mathematics as well as their misconceptions and errors has multiple advantages. First, the TIMSS and TIMSS Advanced assessments have been administered to nationally representative samples of students at regular intervals, starting in 1995 (with the most recent assessments conducted in 2019). Studies using TIMSS data can, therefore, provide a comprehensive picture of student performance in science and mathematics both within and across countries. Second, TIMSS provides the ability to track performance of student cohorts at three grade levels (4, 8 and 12) across multiple assessment years, permitting the evaluation of student performance and misconceptions over time. Lastly, TIMSS and TIMSS Advanced provide access to sets of released items and student performance data from each assessment cycle that can be used for research purposes, such the diagnostic item-level results presented in this symposium, as well as by teachers for instruction and classroom assessment.

Overview of the presentations
There are three presentations: (1) Profiles of U.S. performance in advanced mathematics and physics from the 2015 TIMSS Advanced assessment. This paper provides an in-depth look at U.S. performance on the 2015 TIMSS Advanced assessment using results from secondary analyses conducted in the United States. It examines student performance on items from different mathematics and physics topics to identify areas of relative strength and weakness. (2) Exploring student misconceptions and errors in physics and mathematics using data from TIMSS and TIMSS Advanced. This paper uses 20 years of data from TIMSS and TIMSS Advanced (1995-2015) to explore students’ level of understanding of two core concepts that are introduced in elementary school and further developed through middle and secondary school – gravity in physics and linear equations (or relationships) in mathematics. Diagnostic item-level performance data are used to trace students’ misconceptions and errors related to these concepts across grade levels and education systems in five countries. (3) Using TIMSS and TIMSS Advanced results to inform instruction. This paper illustrates how released TIMSS and TIMSS Advanced items can be a resource for educators in identifying misconceptions and errors, developing diagnostic classroom assessments, and providing focused instruction for students. Using sample items from TIMSS and TIMSS Advanced, practitioners in mathematics and science education discuss specific types of misconceptions and errors related to gravity and linear equations (or relationships) and present example classroom activities to address these misconceptions and errors across grade levels.

Significance
This symposium focuses on applying diagnostic methods to better understand students’ strengths and weaknesses in science and mathematics and further explore specific types of misconceptions and errors related to two core concepts – gravity in physics and linear equations (or relationships) in mathematics. Misconceptions and errors at the upper secondary level are traced to a lack of foundational understanding at earlier grades. Thus, the findings can inform instruction across grades by relating country-level patterns in misconceptions and errors to specific gaps or deficiencies in the curricula or instructional materials. These results from TIMSS and TIMSS Advanced data are a valuable resource for teachers and teacher educators, as teachers who are aware of the types of misconceptions or errors students may make will be able to plan for and provide additional support to their students when they are teaching these or related concepts. Moreover, the methodology described using diagnostic item-level performance data from TIMSS and TIMSS Advanced can be applied to a variety of other topics in science and mathematics. The topics and example items presented in this symposium were selected to illustrate the sorts of findings that can help teachers, researchers, and policymakers better understand student performance. It is hoped this will spur additional study using TIMSS and TIMSS Advanced items in other topic areas.

Structure
The 90-minute session includes three presentations, each about 15-20 minutes. The chair provides a 5-minute introduction and the discussant speaks for about 10-15 minutes after the presentations. For the remaining 15-20 minutes, the chair invites questions from the audience.

Sub Unit

• Large-Scale Cross-National Studies in Education SIG

Chair

• Lydia Malley, National Center for Education Statistics

Individual Presentations

• Profiles of U.S. performance in advanced mathematics and physics from the 2015 TIMSS Advanced assessment - Maria Stephens, American Institutes for Research

• Exploring student misconceptions and errors in physics and mathematics across grade levels using data from TIMSS and TIMSS Advanced - Teresa A Neidorf, American Institutes for Research

• Using TIMSS and TIMSS Advanced Results to Inform Instruction - Alka Arora, American Institutes for Research; Will (Tad) Johnston, American Institutes for Research

Discussant

• Dirk Hastedt, IEA