Linking Quality of Instruction to Instructionally Sensitive Assessments

• In Event: Assessing Transfer of Learning: Instructionally Sensitive Assessments, Curriculum, and Instruction

Abstract

Objectives
This study will evaluate assessment sensitivity with respect to the instructional quality. We claim that a well-developed assessment to measure transfer of learning should be sensitive to the quality of instruction that students are exposed.

Perspectives
We define the instructional sensitivity of assessment as the extent to which the assessment items: (1) represent the intended and enacted curriculum (material covered by the test has actually been taught); (2) reflect the quality of the enacted curriculum; and (3) have formative value. In this study, we focus on the second aspect: instructional quality as it is less studied compared to the aspect of content coverage.

Methods
Our coding system operationalizes the quality of instruction into five main dimensions based on the literature on observations of effective instruction (e.g., Ball & Rowan, 2004; Borko et al., 2006; HRI, 2000; Tarr et al., 2010): (1) alignment to learning targets: addressing the big ideas of the unit or lesson; (2) cognitive demand: openness of teacher questions and type of knowledge involved; (3) sense-making: making connections to ideas from other lessons, real world examples and applications, and other disciplines and support for students to develop in-depth understanding; (4) engagement of students: involving students and valuing students’ contributions; and (5) assessment: effectiveness of formative assessment. We only analyzed the instructional episode of reporting data and drawing conclusions where students are scaffolded to make sense of data to build the explanatory models. Teacher interview and questionnaire data were also analyzed to triangulate the relation between quality of instruction and patterns of students.

Data Sources
This paper presents the empirical evidence of the quality of instruction indicated primarily by video data of 28 5th grade teachers from three science units along with teacher interviews and questionnaires as a source of evidence to support the quality of instruction in order to evaluate the sensitivity claims of test items. We also analyzed students’ scores on the ISAs administered before and after the instruction.

Results
Our preliminary analysis included a series of correlation between the video codes and the effect sizes of pre and posttest for all the items, close items, proximal items, and distal items. Among all the five coding dimensions, only alignment and cognitive demands of learning activities were highly correlated with the effect sizes of close items (r=.74 and r=.84, respectively) whereas the engagement of students was associated with the effect size of proximal items (r=.68). Consistent with our hypothesis, none of these dimensions was correlated with the effect size of distal items. We will further examine the correlation after controlling the instructional coverage and the instructional time.

Scholarly Significance
The study examines the connection between quality of instruction and assessment sensitivity. The findings directly inform educators about the nature and quality of instruction that can lead to gains of student learning at different levels of sensitivity and provide practical recommendations to improve the instructional quality.

Authors

• Ming-Chih Lan, University of Washington

• Min Li, University of Washington

• Maria Araceli Ruiz-Primo, University of Colorado - Denver

• Ting Wang, University of Washington

• Michael Giamellaro, Oregon State University

• Hillary Mason, University of Colorado - Denver