An Examination of [Project Name] and [Project Name]–College Item Features: Differences in Diagnosing Comprehension Difficulties

• In Event: Text Comprehension: What We Know About the Dance Between Reader, Text, and Task in Reading Comprehension

Abstract

Objective
The purpose of this paper is to compare empirically [project name] and [project name]-College holistically and by item features. Such an analysis provides insight to test validity, comprehension development, and influence of text features (e.g., length, genre) on item difficulty by age.

Theoretical Framework
Understanding text requires the coordination of multiple cognitive processes by a reader. Decades of research has provided much awareness about those processes (e.g., strategies, thoughts) which determine how and why some readers struggle with reading comprehension. Yet, traditional reading comprehension assessments generally do not assess the underlying processes that can cause comprehension to falter (Magliano, Millis, Ozuru, & McNamara, 2007), but rather focus on the result (i.e., product) of reading.
There has been recent and continuing pressure to develop measures to identify, diagnose, and address the reading comprehension processing needs of students—both in K-12 schools (e.g., Pearson & Hamm, 2005; RAND Reading Study Group, 2002) and in postsecondary institutions (e.g., Holschuh & Paulson, 2013; Koon & Petscher, 2016). The Multiple-choice Online Causal Comprehension Assessment ([project name] and [project name]) addresses these needs by functioning as a diagnostic tool of specific cognitive processes used by students who struggle with reading comprehension.
By design, both versions of [project name] are diagnostic of student comprehension errors based on known cognitive patterns of poor comprehenders (Authors, 2014, McMaster et al., 2012). [project name] is a sentence-level maze task with diagnostic distractors. Each test item consists of a paragraph with a sentence missing. From three alternatives, the student must select the sentence that best completes the text. [project name] items have three types of responses, one correct response (i.e., a causal coherent inference) and two types of incorrect responses that mimic a common cognitive processes seen from poor comprehenders (i.e., paraphrases, elaborations). Although the basics of the two versions of [project name] are the same, there are notable differences and reasons to examine item features more closely to determine if and how they further moderate comprehension. Examining these features is also important because readers’ comprehension skills are based on combination of abilities (e.g., decoding, fluency), experiences (i.e., background knowledge), goals for reading, and the text itself (e.g., readability, genre), (Kozminsky, 1977; van den Broek, Lorch, Linderholm, & Gustafson, 2001).

Methods & Data Sources
Tests were compared on item features using a combination of human coding (e.g., narrative/expository, emotions), computer software (i.e., Coh-Metrix), and examinee data (i.e., item difficulty, discrimination).

Results and Significance
Preliminary results indicate differences between [project name] and [project name] -College that are by design. Specially, the readability (i.e., Flesch-Kincaid) between the two assessments differs ~4.0 and 9.0, respectively. Moreover, expository text were more difficult than narrative text and had a higher average readability. Additionally, the original [project name] utilized only narrative passages; whereas, approx. 50% of [project name]-College items are expository. Content/genre also varied for narrative items where [project name]-College utilized more realistic fiction than the original. Additional findings to be reported. The results of the analysis and comparison support [project name] (C) as a valid assessment of comprehension processes with limited influence from other text features.

Authors

• Benjamin Seipel, CSU, Chico

• Sarah Elizabeth Carlson, Georgia State University

• Mark L. Davison, University of Minnesota

• Gina Biancarosa, University of Oregon

• Virginia Clinton-Lisell, University of North Dakota

• Patrick Kennedy, University of Oregon