Reliability of Aggregation of Results on Multiple Assessments: Which Role Can Retests Play?

• In Event: Roundtable Session 117
  In Roundtable Session: The Whole Is Greater Than the Sum of Its Parts: Research on Aggregated Test Scores

Abstract

Guidelines in educational measurement, e.g., the Standards for Educational and Psychological Testing (AERA, 1999), underline that high stakes decisions on students should not be based on a single score. Consequently, multiple test scores are combined into a final decision (Chester, 2003). Van Rijn, Béguin and Verstralen (2009) as well as Douglas and Mislevy (2010) have shown that the way the scores are combined can have major impact on the reliability of the decision. Douglas and Mislevy (2010) have also shown the influence of the number of retests on classification accuracy. However, they have only considered a scenario where five tests were combined and every student that failed could take one or two retests on all courses. The purpose of this research is to investigate in more depth the influence of retests on the reliability of decisions based on multiple assessments.

The following questions will be answered:
- Who has to take retests? Those who failed at the first assessment, or those who have a substantial probability of getting another outcome when the total assessment would be repeated?
- On how many and on which tests do these students have to take retests?
- Which score should be taken? The best or the mean score over retests?

First, the method that was proposed by Douglas and Mislevy (2010) is used in order to demonstrate the influence of the selection of students, the selection of retests, and the method to combine scores on the reliability of the decision. Second, an IRT based Plausible Value algorithm will be proposed for optimal selection of students for retests. This method quantifies for each student the probability of obtaining another outcome in a replicate the total assessment. This will be demonstrated with data taken from national secondary education examinations.

The results show that optimal selection of students for retests does improve the reliability more than selecting only students that failed on the first administration. However, this effect is small in cases with only one retest. The results also show how in some cases optimal selection of a small percentage of the students can account for the vast majority of reliability improvement that would have been obtained when the tests would have been lengthened for all students. Further research will elaborate on multiple shorter retests, and on retests that fit at the ability level around the cut score, i.e., multistage testing.

Author

• Robert Zwitser, Cito