Measuring Students' Argument Literacy Skills in Speaking, Listening, Reading, and Writing

• In Event: 53.026 - Broadening Opportunities for Classroom Learning With Dialogic Pedagogies

Abstract

In this poster, we describe a systematic process of developing and validating a set of measures designed to assess elementary school children’s argument literacy. We define argument literacy as the ability to comprehend, formulate, and evaluate arguments in speaking, listening, reading, and writing. Our measures represent open-ended, scenario-based assessments that include an oral interview, a reading task, and a written essay.

Purpose and Theoretical Framework
Major educational frameworks today view argumentation and related skills as an important outcome of schooling, suggesting that these skills should be targeted across grade levels and school subjects (National Governors Association Center for Best Practices & The Council of Chief State School Officers, 2010). Unfortunately, the widespread recognition of the value of argumentation has not generated enough efforts to understand how argumentation skills can best be measured, especially in the elementary grades.

In this poster, we discuss our work on developing measures of argument literacy that are theoretically grounded, psychometrically sound, and practical for use in a typical elementary school classroom. Our understanding of the construct of argument literacy integrates schema-theoretic notions proposed by educational psychologists (Anderson & Pearson, 1984) and the normative models of argumentation developed by philosophers (Walton, Reed, & Macagno, 2008). We suggest that knowledge of argumentation can be represented as a generic mental structure that we call an argument schema. A well-developed argument schema should include the understanding of key standards related to argument quality: 1) consideration of alternative perspectives, 2) clarity of arguments language and structure, 3) acceptability of reasons, and 4) the logical validity of arguments. These standards guided the development of our measures of argument literacy.

Methods and Data Sources
We followed a systematic, multi-step process of test construction proposed by Crocker and Algina (1986). We began with a comprehensive literature review to identify relevant behaviors and to formulate hypotheses to be tested in validity studies. Next, we created an initial set of measures and engaged in an iterative process of piloting and revising the measures. Over the next two years, we administered the initial and revised measures to 393 students. We developed the scoring protocols and made revisions based on examination of student performance. Additional information to inform revisions came from individual and focus group interviews with 65 students and 9 teachers. We administered our final measures to 656 students from 34 fifth-grade classrooms. Multiple raters scored student performance using the revised protocols, and we conducted multiple analyses to examine the psychometric properties of the measures.

Results and Significance
We collected evidence to support high inter-rater reliability for all measures of argument literacy, with inter-rater reliability coefficients ranging from .82 and .95. We also examined student performance in relation to a variety of demographic, academic, and motivational variables, including gender, ethnicity, reading comprehension, and topical interest. Our results indicate that all three measures of argument literacy provide reliable and useful information about student performance that can inform classroom instruction and research on supporting the development of argument literacy skills of elementary school students.

Authors

• Alina Reznitskaya, Montclair State University

• Ian A. G. Wilkinson, Ohio State University