Comparing Two Experimental Designs to Evaluate Item Sensitivity

• In Event: Assessing Transfer of Learning: Instructionally Sensitive Assessments, Curriculum, and Instruction

Abstract

Objectives
We will explore methods for evaluating whether items developed in the project are sensitive to instruction, using uninstructed control groups. We will perform three comparisons: (1) pretest and posttest performance for 5th graders, (2) item gains for uninstructed 5th graders in repeated pretesting to gains for the same students after instruction (posttest), and (3) results for uninstructed 4th graders to pretest and posttest results for 5th graders.

Perspective
Improvement in item performance may result from development or familiarity with the test, rather than sensitivity to instruction. We examine these issues by comparing instructed 5th graders with two types of control conditions: their own pre-instruction scores, and a 4th grade control group.

Methods
We administered the ISA test on the Landforms unit to 5th graders three times: twice prior to module instruction, and once after. The two pretests can be considered as an uninstructed group. We administered the same test to 4th grade students who were not taught the module.
We examined student performance on 20 items. For each item, we performed three statistical tests comparing the post-instruction gain of the 5th grade students with the defined un-taught groups. First, we performed a t-test to compare the pre-instruction (pretest 2) vs. post-instruction (posttest) performance for 5th graders. Second, we conducted a t-test to determine whether the gain after instruction (i.e., from pretest2 to posttest) was larger than the gain of pre-instruction (i.e., from pretest1 to pretest2). Third, we performed two t-tests to compare the performance for 4th graders (pre-instruction) to the performance for 5th graders of pre- and post-instruction.

Data Sources
The data analyzed in this poster were student item level scores in the administered ISAs.

Results
The tests of pre-instruction vs. post-instruction gain for 5th graders showed some differences from the comparison of pre-test vs. post-test performance. For three items, posttest performance differed significantly from pretest, but post-instruction gain did not differ significantly from pre-instruction gain. In other words, performance gain was similar before and after instruction. This underscores the use of the repeated pretest design rather than a simple pre and posttest design as it allows identifying items on which the improved student performance is more a result of factors other than instruction (e.g., familiarity with the test).

We compared 4th graders as the pre-instruction group with the 5th graders as the post-instruction group. The comparison with 5th graders’ pretest showed no significant differences between the two groups except one item, suggesting development (4th grade to 5th grade) had a minimal effect for this test. As predicted, significant differences were identified between 4th graders and the posttest of 5 graders, which can be considered as evidence supporting the sensitivity claims of items.

Scholarly Significance
Polikoff (2010) notes that a pre and posttest design without a control group raises questions about internal validity, since improved performance may result from factors other than instruction. We empirically compare two strategies for incorporating the control condition in the pre and posttest design and our findings illustrate the complexity in the comparison methods.

Authors

• Kellie Wills, University of Washington

• Min Li, University of Washington