Teachers’ and Students’ Perceptions about Instructionally Sensitive Assessments: Disentangling the Meaning of Transfer of Learning

• In Event: Assessing Transfer of Learning: Instructionally Sensitive Assessments, Curriculum, and Instruction

Abstract

Objectives
This poster will focus on teachers’ and students’ perceptions on each of the instructional sensitivity dimension of assessment items. The research questions were: What is the agreement between item developers, teachers, and students about the level of instructional sensitivity of the items? What factors influence the sensitivity perceptions by teachers and students? Do teachers find the information provided by an item useful to make adjustments in their teaching? Evidence about the validity of items to the instruction students’ received, its quality, and its formative value will be provided.

Perspective
We define instructional sensitivity of assessments as whether the assessments represent the intended and enacted curriculum, reflect the quality of the enacted curriculum, and have formative value. Consistent with the social validity notion of Wolf (1978), analysis of teachers’ and students perspectives provides judgmental evidence on the sensitivity claims of the items. We hypothesize that their perceptions can be largely influenced by their own implementation of curriculum, background, and beliefs.

Methods
Qualitative analysis was employed to examine the interview transcriptions and questionnaire data regarding whether teachers and students held the similar perceptions of item sensitivity as researchers and item developers as well as factors that influenced their perceptions.

Data Sources
Twenty eight teachers and 207 students, nine students randomly selected from each classroom, covering three units. Various data were collected about their judgment and reasoning of item sensitivity. Teachers were asked to complete a questionnaire about every assessment item administered to their students and participate in a semi-structured interview and a focus group. Students participated in a think-aloud protocol with different sets of three or four assessment items, and responded to a semi-structured interview on each of the items.

Results
Selected findings include:
* Teachers’ classification of item sensitivity was the same as that defined by researchers for only 73.17% of items.
* Teachers and students agreed as to whether the content of an item was taught only 54% of the time.
* Teachers tended to see proximal items as distal. Their ideas about TOL were different than the item developers. Teachers’ conceptions of TOL were influenced by very small changes made in the item from what the curriculum prescribed (e.g., a word – container to cup, data tables including two additional data points).
* Students tended to see proximal items as proximal. They were able to “see” the transfer of what they learned more easily than their teachers.
* Item sensitivity and difficulty seemed intertwined when teachers attempted to classify items on the close-to-distal continuum. In contrast, students’ perceived item sensitivity was influenced mainly by the degree to which an item “made me really have to think.”
Upon seeing their students’ assessment results, a majority of teachers indicated they would use those data formatively to inform future instruction.

Scholarly Significance
This poster will provide detailed analysis of data from different sources, in a triangulated manner. The findings will expand the conceptualization of assessment sensitivity by considering and evaluating the role of teachers’ and students’ perspectives in validating ISAs.

Authors

• Michael Giamellaro, Oregon State University

• Deanna J. Sands, University of Colorado - Denver

• Kellie Wills, University of Washington

• Jennifer Feehan, University of Colorado - Denver

• Maria Araceli Ruiz-Primo, University of Colorado - Denver

• Min Li, University of Washington