Consequential Validity Evidence for an Alternate Science Assessment

• In Event: Alternate Assessments: Looking Back, Looking Forward, Looking Around

Abstract

The examination of validity evidence for large scale assessments encompasses many forms, and examining consequences of assessment programs is one recommended aspect of validity arguments (Kane, 2001; Kane 2006, Kane, 2010; Lane, Parke & Stone, 1988; Lane & Stone, 2002; Messick, 1989).
The purpose of this study was to examine the consequential validity of one statewide alternate science assessment. The study incorporates two main sources of evidence. First, in order to gain an understanding of the science content, instructional methods, depth of knowledge, and degree of linkage to statewide achievement standards of classroom activities used to teach science to students with significant cognitive disabilities, classroom artifacts were collected and coded.
Second, for three consecutive years, participants at the scoring sessions for the science portion of a statewide AA-AAS, teachers of science to students with significant cognitive disabilities, were administered a questionnaire asking about their science teaching responsibilities, the impact of the statewide assessment system on their instructional and assessment practices, and the content and methods of their instruction.
Participants for the classroom artifact study also included teachers who volunteered and were selected to participate in scoring activities for a statewide AA-AAS. These teachers provided examples of classroom artifacts including lesson plans, instructional activities, and instructional resources that were analyzed in the current study. Approximately 50 teachers provided a total of 157 artifacts. Along with the artifacts, teachers completed a basic descriptive questionnaire about each artifact, which included information such as the type of activity, who developed the activity, the science content addressed by the activity, the type of science class, and grade level in which the activity was used. In addition to this teacher-provided descriptive information, each artifact was coded by a panel of experts consisting of experienced teachers of students with significant cognitive disabilities. The coders were trained and coded the classroom artifacts based on several criteria including type of science, alignment to state alternate and general content standards, depth of knowledge, functionality level and age appropriateness.
Results showed that overall, overall, 74% of the artifacts were rated as aligning to the statewide AA, and 92% of the artifacts were considered to be science, as opposed to not science or foundational. Higher percentages of artifacts were found to align to the alternate standards than the regular education standards. In terms of depth of knowledge, the majority of activities addressed Level 1 (42%) or Level 2 (40%), but some activities met the higher depth of knowledge ratings. The content area of biology was addressed by the most artifacts (41%), but nature of science (13%), physical science (25%) and earth and space science (21%) were also addressed. In the full paper, detailed results will be presented overall and separately for high school, middle school and elementary levels.
Participants for the survey research study also participated in the scoring sessions of the statewide AA-AAS. Between 100 and 200 participants were surveyed each year. Descriptive results summarizing the findings for individual years and trends over time will be presented.

Authors

• Peter W. Heh, University of Pittsburgh

• Mary A. Hansen, Robert Morris University

• Steven R. Lyon, University of Pittsburgh

• Naomi K. Zigmond, University of Pittsburgh