Qualitative and Quantitative Feature Analysis of Fourth-Grade U.S. Tests

• In Event: 64.036 - Feature Analysis Approaches in International Assessments

Abstract

Objectives. The objectives were: 1) to describe the features of items found within and across the 4th grade English Language Arts (ELA) and Mathematics tests; 2) conduct a quantitative analysis of the feature-related data yielding so as to provide insights into item variance and item difficulty; 3) to compare findings with the international presenters.

Perspectives. An item or task might be rated as more difficult due to high cognitive load, rich content, and narrative text, or in contrast, easy because to a simpler target concept or shorter task and response requirements but these are assumptions. We used a systematic qualitative technique, i.e., feature analysis to rate features, e.g., cognitive demands (problem solving or declarative knowledge), task requirements (representation, symbol systems and response mode). Finally, specific content is rated. Our larger intentions are to use features to confirm test specifications or descriptions, to predict impact on student performance, and to report to test designers and practitioners the effects found in data. The same features can be rated and analyzed across countries giving a parsimonious tool to ensure alignment among instructional modes, assessment development, student learning, to compare assessments, and to conduct novel validity analyses.

Methods. We use both qualitative and quantitative analyses to derive conclusions. Following ratings, assessment items or tasks were tagged with features. Next, quantitative analyses of student performance were used to determine the relationships among quantitative features and assessment performance.

Data. Both ELA and Math (25 items each) were analyzed. All of these items were computer adaptive test items, and the sample size of ELA and math were, respectively, 35,689 and 12,651.

Results. Interrater reliabilities of feature ratings were 0.90 and 0.95, for ELA and Math, respectively. For math items, four math concepts (i.e., number representation/conversation, units and currency, decimals/place value, and simple equation) significantly increased item difficulty, while one math concept (i.e., fractions) appeared to significantly decrease item difficulty. Math vocabulary significantly increased item difficulty, and items with equations were significantly easier. Problems requiring students to decipher charts and graphs were more difficult. These results are consistent with our cognitive labs study results. With respect to problem type, problem solving items with procedural/algorithmic thinking appeared more difficult items than items with recall and reproduction. For ELA items related to language complexity, passive voice and complex sentences significantly increased item difficulty, while both interrogative sentences and passage reliance contributed to reduce item difficulty. As to text type, single sentence prompts unexpectedly related to increased item difficulty. In terms of item formats, items with drag and drop made items more difficult than multiple choice or multi-select items. In terms of comprehensive types related to cognitive processing, literal or interpretive comprehension type items found significantly easier than the items requiring inference or application.

Scientific Significance. Mixed mode analyses of tests can identify contributions to difficulty made by item features, which, in turn have implications for using results to characterize and improve instruction, test design, or to draw inferences across tests about the generalizability of the findings.

Author

• Kilchan Choi, University of California - Los Angeles