Investigating Effects of the Complexity of Leveled Texts on Student Comprehension

• In Event: 73.020 - Examinations of the Construct of Challenge: Readers, Texts, and Tasks

Abstract

PERSPECTIVE: Leveled passages where students read progressively more complex texts are frequently used in text-referenced assessments (Pearson & Hamm, 2005) to determine student reading levels, reading comprehension and fluency. Despite their popularity as classroom assessment and progress monitoring tools, little rigorous empirical evidence is available for the validity of the leveling of these assessment passages and their associated items.

Typically, assessment developers report on readability scores (e.g., Lexile) based on the analysis of text features alone. Some developers show descriptively that student performance (in raw scores) declines as the level of text increases to show the progression of text levels. However, these methods fall short on more fully investigating whether readability-based prediction of text leveling is empirically supported by student performance and what makes passages and associated items more difficult for which type of students.

DATA & METHOD: This study used item-response theory (IRT) models to investigate the validity of testlet leveling. A testlet in this study was comprised of a passage and five comprehension questions about the passage. This study included 48 testlets from a web-based program designed primarily for grades 3 and above. The testlets are organized into a 12-level system, corresponding to grades 1-12. All 48 passages in the testlets were carefully developed to meet grade-specific text complexity scores in Lexile. Passage writers ensured that text complexity was achieved through careful, systematic control of vocabulary and semantic difficulty, rather than on simplistic manipulation of sentence length and syntactic features. Similarly, the comprehension questions were developed according to a research-based rubric, ensuring each passage would have parallel items.

RESULTS: In the first round of analyses, item scores were summed at the passage/testlet level and a partial credit Rasch Model (Masters, 1982) was applied to obtain estimates for the difficulty of the 48 testlets. Results indicated that, in general, the 48 testlets had the intended progression in difficulty as the level of the testlets increased. A simple regression analysis showed that the developer-determined levels were statistically significant predictor of the estimates of testlet difficulty (b = 0.06, t(46) = 7.82, p < .000 ) and that these levels explained 57% of the variance in the estimated testlet difficulties. Interestingly, in separate regression models, neither Lexile’s overall scores nor its subscale scores (i.e., the log of mean sentence length and the mean of the log word frequency) were significant predictors of the testlet difficulty estimates. Further analyses with more sophisticated IRT models are currently underway to identify contributions of passage features as well as item features that constitute to the testlet difficulty estimates. Additionally, differential passage/testlet functioning among subpopulations (e.g., students with vs. without prior knowledge of passage topic) are being analyzed.

IMPLICATIONS: This study demonstrates an approach for validating passage/testlet leveling for test developers. It also offers methods for empirically investigating text complexity, taking into account characteristics of passages, items, and readers.

Authors

• Yukie Toyama, University of California - Berkeley

• Alexandra N. Spichtig, Reading Plus/Taylor Associates