Evaluating the Validity of Instructionally Sensitive Assessments

• In Event: Validity of Educational Assessments: Capturing (Instructional Effects on) Students' Learning Growth

Abstract

Objective
This paper presents a study on developing and evaluating an approach for constructing instructionally sensitive assessments. We define instructional sensitivity as the extent to which assessment items: represent the planned curriculum material covered by the assessment; reflect the quality of instruction; and have formative value to adjust instruction. In the paper we focus on the first two aspects by providing validity evidence on two proposed interpretations of instructionally sensitive assessments: students’ observed performance in instructionally sensitive assessments reflect their knowledge as a consequence of instruction; and students’ performance on instructionally sensitive assessment reflects, at least partially, the quality of the instruction received.

Theoretical Framework
Following Ruiz-Primo, Shavelson, Hamilton, and Klein (2002), we conceptualize instructional sensitivity as a continuum with different distances to the planned and the enacted curriculum. Assessments then can be: immediate such as artifacts from the enacted curriculum; close tapping similar content and activities described in the planned curriculum and implemented in the classroom; proximal tapping knowledge and skills relevant to the curriculum, but with contexts (e.g., scenarios) differing from those studied in the curriculum; and distal tapping state or national standards in a particular domain rather than a specific curricula that can be potentially enacted in a classroom. In the parent project we developed an approach to constructing assessments at different distances of curricula based on the manipulation of the characteristics of the items.

Methods
To test the robustness of this approach three science modules at Grade 5 were used, one BSCS module: Heat and Change, and two FOSS modules: Landforms and Environments. Each module was linked to a different school district. Information about teacher and student participants is provided in Table 1.
Table 1. Information about Participants

Each classroom was videotaped during the entire implementation of the module. These videos were coded on different aspects to capture information about the quality of instruction in the classroom, such as communicating of learning goals to students, supporting students to making conceptual connections, instructional activities are aligned with the learning goals, and discussion of questions. In addition, students were administered the assessment items varying in instructional sensitivity, using the pretest-and-posttest design. Both video codes and students’ scores are used to provide evidence to test the two proposed score interpretations regarding instructional sensitivity.

Results
We will describe the assessment development approach that can directly improve how evaluation studies are conceptualized. We will focus on using characteristics of the modules as manipulation in item components. Patterns of item manipulation provide evidence about the consistency of the manipulation across modules to determine the proximity of the items (e.g., the context of the item). The judgmental and the empirical evidence suggests the approach is promising. Consistent with our hypothesis, the patterns of the effect size corresponded to the manipulation with respect to item proximity. Such patterns were replicated across modules.

Scholarly Significance
We seek to support the research agenda around the development, study, and use of instructionally sensitive assessments.

Authors

• Maria Araceli Ruiz-Primo, Stanford University

• Min Li, University of Washington