Instructional Sensitivity and Transfer of Learning at Different Distances: Close, Proximal, and Distal Assessment Items

• In Event: Assessing Transfer of Learning: Instructionally Sensitive Assessments, Curriculum, and Instruction

Abstract

Objectives
This poster focuses on the approach used to develop instructionally sensitive assessments; more specifically, on the curriculum characteristics that allowed the measure of transfer of learning (TOL) at different distances by developing close and proximal items. The paper is guided by two research questions: What module characteristics can be systematically manipulated to develop items at different distances (close and proximal) that prove to be instructionally sensitive? What characteristics should items tapping different levels of transfer of learning have?

Perspective
To understand the learning goals of the science module we proposed two activities during the mapping: (1) identify the scientific knowledge and scientific practices students should achieve if the science unit is taught as intended by curriculum developers. And (2) classify these scientific knowledge and scientific practices by types of knowledge - declarative, procedural, or schematic. The key question asked for each lesson of the unit was, “What are the critical concepts, procedures, processes, explanations, or principles to which teachers and students need to pay attention?” We hypothesize that some characteristics of any science module can be potential sources of manipulation of instructional sensitivity to measure TOL.

Methods
We mapped three science modules on seven aspects for each lesson: (a) the learning targets in terms of scientific knowledge and scientific processes; (b) the type of knowledge students are engaged (i.e., declarative, procedural, and schematic knowledge); (c) the activities that are critical to achieve the learning targets; (d) the documentation required of students; (e) the materials used; (f) the graphical representations; and (g) the scientific vocabulary involved in activities/investigations. These aspects were analyzed within the big ideas identified by module.
Using the big ideas and learning targets as a guide, the item development was carried out on the basis of “bundles of triads.” For each bundle, one close item (C) was developed first and then two proximal items (P1 and P2) were developed using the close item as a reference. The difference between the two proximal items was the extent of the deviation from the close item: minor changes for P1 and major changes for P2.

Data Sources
Information about students’ performance across the items at different distances was collected in a pretest-posttest design.



Results
Manipulations to the bundles were tracked for each module. Across the units, we observed that: (1) close items show higher percentage in “no change” and Proximal 2 items showed higher percentages of “big change,” which was aligned with the item development rules for manipulating. (2) Manipulations of certain item characteristics based on the maps (e.g., type of organism, type of setting/system) are more suitable to measure the transfer of student learning than other item characteristics.

Scholarly significance
The paper will provide: (1) information about an assessment development approach that can directly improve the way that teachers, curriculum developers, and evaluators develop and conceptualize assessments with illustrative examples; (2) empirical evidence on the importance to consider instructional sensitivity as an aspect of validity critical to the evaluation of the effectiveness of instruction or instructional programs.

Authors

• Min Li, University of Washington

• Maria Araceli Ruiz-Primo, University of Colorado - Denver

• Michael Giamellaro, Oregon State University

• Kellie Wills, University of Washington

• Hillary Mason, University of Colorado - Denver

• Jennifer Feehan, University of Colorado - Denver