Instructionally Sensitive Assessments Across Three Science Units

• In Event: Assessing Transfer of Learning: Instructionally Sensitive Assessments, Curriculum, and Instruction

Abstract

Objectives
This poster will examine how the patterns of instructionally sensitive assessments (ISAs) across the three tested science units can be explained by the curricula variation. We will tackle the following research question: How do curriculum variables influence the patterns of student performance on ISA items?

Perspective
Instructionally sensitive assessments can be developed to measure the extent to which student learning has transferred. Differentiated patterns of student performance should correspond to item sensitivity. In this poster, we explore the curriculum variables that influence such performance patterns. Applying the curriculum mapping approach, we can characterize the instructional activities that are supposed to accomplish the identified learning objective. The characteristics describe the nature of learning experiences assumed by the curriculum developers, such as type of activity (e.g., defining terms, designing the experiment, drawing conclusions), social context, involvement of students, and quality of guiding questions. We hypothesize the nature of activities can influence the patterns of student performance since different types and uneven quality of the activities present learning experiences varying in quality.

Methods
Based the item level scores of ISAs, we generated effect size (ES) estimates to index the pretest-posttest gain of student learning. We used two types of estimates: (1) sensitivity as close, proximal, and distal; (2) type of learning as content and science process skills. We then examined the curriculum codes by considering patterns of the learning gains.

Data Sources
We analyzed two sources of data: scores of the ISAs from the three science units, and codes of instructional activities derived from the curriculum maps for each unit.

Results
Regarding the assessment sensitivity, the ES estimates of student performance for the three units are 1.07, 0.88, and 0.63 for the close, proximal, and distal items for Heat and Change unit; 1.00, 0.52, and 0.26 respectively for Environments unit; and 1.35, 0.95, and 0.04 respectively for Landforms unit. As predicted, the amount of learning gains decreases as items becomes less sensitive to the intended curriculum. Interestingly students show more sizeable gain on content learning than on the process skills, as 0.10-0.22 larger for the ES estimates across the three units.

We further examined the curriculum codes against the patterns of these effect sizes. For example, the guiding questions for drawing conclusions in Heat and Change and Landforms units are more closely tied back to the big ideas and patterns with better quality than those in the Environments, therefore the two units had relatively larger effect size on the proximal items. The investigations in Landforms tend to be more robust and results are easier to observe or measure and reliable which then led to larger gain of student performance on close items which are similar to the activities students had learned.

Scholarly Significance
The empirical findings offer advice to curriculum developers and educators about the instructional characteristics that can influence the student learning. This new understanding will lead to a greater clarity on instructional activities to support the learning transfer.

Authors

• Min Li, University of Washington

• Maria Araceli Ruiz-Primo, University of Colorado - Denver

• Michael Giamellaro, Oregon State University

• Kellie Wills, University of Washington