PISA 2015: Cross-Mode Comparison Between Paper- and Computer-Based Assessments

• In Event: Measurement and Methodological Challenges and Treatments in Large-Scale Database Analyses

Abstract

This study uses 57 dichotomous mathematics items from the Program for International Student Assessment (PISA) to investigate differential item functioning (DIF) between the computer-based assessment (CBA) and the paper-based assessment (PBA) across six countries: Argentina, Indonesia, Peru, Romania, Thailand, and Turkey. First, we examined whether parameter-invariance was achieved for both test modes. Then, we investigated if mode effect exist using Mantel-Haenszel and logistic regression DIF methods. The findings indicated possible mode effect. In addition, logistic regression DIF method flagged more noninvariant items than Mantel-Haenszel statistic. However, both methods agreed that 16 out of 57 items did achieve measurement invariance across countries. In contrast, there were 28 items flagged by both methods as evidencing DIF, and recommended for additional review.

Authors

• Nnamdi Chika Ezike, University of Arkansas at Fayetteville

• Wen-Juo Lo, University of Arkansas