The Impact of Student Characteristics on Estimates of Test Items' Instructional Sensitivity

• In Event: Validity of Educational Assessments: Capturing (Instructional Effects on) Students' Learning Growth

Abstract

Instructional sensitivity is the psychometric property of a test or an item to capture effects of the content and quality of instruction (Polikoff, 2010). The assumption that outcome measures are instructionally sensitive is widespread in education, but is rarely empirically investigated (Naumann, Hochweber, & Hartig, 2014). However, evaluating instructional sensitivity is of crucial importance, because misinterpretation of test results may lead to wrong inferences regarding instruction (Popham et al., 2014).

Evaluations of instructional sensitivity usually focus on direct effects of instructional measures on test scores or item parameters. However, according to utilization-of-learning-opportunities models, students’ learning progress in school depends on the opportunities to learn (OTL) provided by instruction and the utilization of OTL by the students (Helmke & Schrader, 2016). Moreover, the utilization of OTL depends on cognitive, motivational, and volitional student characteristics. Consequently, we assume that current measurement of instructional sensitivity is oversimplified. Thus, our study aims at the extent to which student characteristics influence indicators of instructional sensitivity.

We use data from a Swiss mathematics study. The sample comprises 951 fifth-graders from 48 classes. The assessment comprised 45 arithmetic tasks administered at four time points throughout the school year 2016/2017. Student characteristics (e.g. Rakozcy, Buff, & Lipowsky, 2005) and instructional quality (e.g. Klieme, Pauli, & Reusser, 2009; Pianta & Hamre, 2009) were assessed via questionnaire using well-established scales.
We analyzed instructional sensitivity using the longitudinal multilevel-IRT (LMLIRT) approach (Naumann, Hartig, & Hochweber, 2017). This approach provides estimates of classroom-specific initial item difficulty and change of difficulty across time as measures of instructional sensitivity. In a first step, we investigate to which extent change in item difficulty is attributable to instruction. Then, we examine whether change in item difficulty is affected by student characteristics. Finally, we investigate the interaction between instruction and student characteristics.

Results indicate that instructional quality explains a considerable amount of variance in some items’ difficulties, while other items appear unaffected (e.g. supportive classroom climate: range of explained variance of change in item difficulty (R²): 0% to 42%; meanR²=6%). However, the effects aren’t significant for most items (95% Bayesian credible interval comprised zero). By including student characteristics in the model, effects of instructional quality on change in item difficulty decrease or vanish (e.g. supportive classroom climate in consideration of extrinsic motivation through identification and introjection: range of explained variance of change in item difficulty (R²): 0% to 52%; meanR²=18%). Yet, similar to results above, most effects are not significant. However, we found interaction effects of instructional quality and student characteristics in some items.

In summary, our analyses indicate that both the provision of OTL and its utilization influence the instructional sensitivity of test items. There are items which are affected by instructional quality and/or student characteristics. Yet, numerous items are not despite a considerable amount of variance is explained by both predictors. Accordingly, neglecting student characteristics may lead to wrong inferences on instructional sensitivity. In our contribution, we will discuss the role of students’ characteristics with respect to instructional sensitivity.

Authors

• Stephanie Musow, University of Teacher Education St. Gallen

• Alexander Naumann, German Institute for International Educational Research

• Jan Hochweber, University of Teacher Education St.Gallen

• Johannes Hartig, German Institute for International Educational Research