On the Sensitivity of Achievement Test Items to Cognitive Activation

• In Event: Explaining Teaching Effectiveness Beyond Value-Added Models

Abstract

Assessments of students’ achievement are widely used in educational research and policy (Pellegrino, 2002). Yet, inferences on teaching and teachers drawn from students’ test scores can be sensitive to the ways in which achievement has been measured (Grossman, Cohen, Ronfeldt, & Brown, 2014). Consequently, many researchers demand that valid interpretations of results require more detailed information about how tests and items are capable of capturing effects of classroom instruction, that is, instructional sensitivity (IS; Popham, 2007).
While IS of tests is commonly investigated by regression of test scores on measures of instruction, current analyses of items utilize purely statistical approaches (Polikoff, 2010). However, knowledge on these sensitivity indices’ relationship to instruction is scarce. Thus, the present study aims at examining how students’ responses on items are affected by classroom process quality, particularly cognitive activation (CA; Klieme, Pauli, & Reusser, 2009; Pianta & Hamre, 2009). CA comprises teaching practices that challenge students’ understanding, for example, by discussing students’ answers and confronting them with conflicting ideas. Teachers who cognitively activate their students create challenging classrooms in which students are actively involved in learning.
To quantify the IS of items, we build on a recent IRT approach to estimate classroom-specific change in item difficulties between a pretest and a posttest (Naumann, Hochweber, & Hartig, 2014). The model provides two sensitivity indicators: the average change in classroom-specific item difficulty across time points (global sensitivity), and the variation of change in item difficulty across classes (differential sensitivity). We adapted the model to incorporate individual and classroom-level predictors for initial values of and change in classroom-specific item difficulties, thus investigating the degree to which an item’s differential sensitivity actually corresponds to IS.
Data used for analyses comprised responses of about 1070 3rd-graders in 54 classes of German elementary schools on fifteen items measuring students’ conceptual understanding that were common to pre- and posttests framing a curriculum on floating and sinking. We used video-based or on-line ratings on CA in class as predictor in latent regression, controlling for students’ individual background variables. Analyses were carried out in a Bayesian framework using MCMC estimation.
Results showed that six items’ difficulty at T1 and three items’ change in difficulty between T1 and T2 was significantly related to CA. This provides indication that about two third of the items may be classified as instructionally sensitive with respect to the degree of CA in class. Furthermore, IS of the tests decreased when items sensitive to CA were excluded and increased when insensitive items were excluded.
Information about items’ IS are highly relevant, as it allows, among others, to purposefully select items that fit the intended area of application. If one is interested in the effects of teacher or classroom characteristics on student achievement, implementing items that reflect differences with respect to these characteristics appears necessary (high IS). On the other hand, in testing of individual abilities, items with less sensitivity towards teacher or classroom characteristics may be preferred, to allow for a clear attribution of results to individual differences (low IS).

Authors

• Alexander Naumann, German Institute for International Educational Research

• Svenja Rieser, Goethe University

• Jan Hochweber, German Institute for International Educational Research

• Johannes Hartig, German Institute for International Educational Research