Association Between Readability of Mathematics Word Problems and Performance on the NAEP and TIMSS (Trends in International Mathematics and Science Study)

• In Event: 35.020 - Excellence in Education Research: Early-Career Scholars and Their Work

Abstract

Educational research in the United States has suggested that mathematics word problems are notoriously difficult (Cummins, Kintsch, Reusser, & Weimer, 1988); however they are prevalent in curriculum, instruction, and assessment (Jonassen, 2003). The performance of U.S. students on international assessments like the Programme of International Student Assessment (PISA) and the Trends in International Mathematics and Science Study (TIMSS) highlights U.S. students’ difficulties with story problems. The PISA, for example, accentuates using math in real world situations that arise in the workplace, society, and everyday life (Organisation for Economic Cooperation and Development [OECD], 2013), and the U.S. has performed below the OECD mathematics average since the first administration in 2000.
In the present study, we examine factors of difficulty of mathematics word problems that are related to the problem’s verbal language – the nonmathematical characteristics like number of sentences, cohesiveness of the text, real world topic (e.g., whether the problem is about farming or banking), and use of pronouns. We use two computerized text analysis tools to analyze the readability and topic of mathematics word problems from the NAEP and TIMSS, and relate these factors to student performance. We also examine how the effects of these readability and topic measures varies based on student background characteristics including race, language, and socio-economic status.
Our study included the 4th and 8th grade released mathematics items from the NAEP 1990-2013, and the TIMSS 1995-2011 – this encompassed 757 NAEP problems and 445 TIMSS problems. We analyzed the text of the word problems using LIWC (Pennebaker, Booth, & Francis, 2007) and Coh-Metrix (Graesser, McNamara, Louwerse, & Cai, 2004). LIWC comprises various dictionaries of words in different categories, such as auxiliary verbs or “leisure” words; its output consists of the percentage of words used from each dictionary. Coh-Metrix analyzes cohesion relations and measures of language, text, and readability, such as word frequency, parts of speech, and syntactic complexity.
Data were analyzed using mixed effects linear regression models (on datasets that had average student performance on each problem), and mixed effects logistic regression models (on datasets that were broken down to show each students’ response to each question) (Snijders & Bosker, 1999). The dependent measure was the accuracy of U.S. students on each problem, either measured as an overall percentage or as a 0/1 correct/incorrect. On the NAEP, we found a statistically significant decrease in performance associated with more sentences in problems, with second person pronouns, and with the use of vocabulary words acquired later in life. Pronouns in general are associated with significantly higher performance. On the TIMSS, number of sentences no longer has a significant relationship with performance, but second person pronouns are again associated with lower performance. In addition, causation words (e.g., why, how, reason) are associated with lower performance on the TIMSS. The effect size for the readability/topic measures (Ω2) is small – the proportion of variance explained was 5-7%.

Authors

• Candace A. Walkington, Southern Methodist University

• Virginia E. Clinton, University of North Dakota