From cheating to learning: An evaluation of fraud prevention on national exams in Indonesia

• In Event: From where you are to where you want to be: How information shapes goals, behaviours and motivation of actors in education

Proposal

Cheating reduces the signal value of exam data and it might shift the focus of teachers and students away from learning. However, it is difficult to prevent cheating if it is widespread. In Indonesia, many teachers and students shared answers sheets and helped each other during the exam. In 2015, the Government of Indonesia developed a so-called ‘integrity index’. The index is based on previously developed algorithms that detect suspicious patterns on answer sheets. It shows evidence of substantial cheating on the national exam in 2015 for about a third of all junior secondary schools (grades 7-9).

We evaluate the impact of computer-based testing (CBT) on national exam scores in junior secondary schools in Indonesia, exploiting the phased roll-out of the program from 2015 to 2019. The CBT program draws exam items directly from a server, such that each student has a unique test version. This way, answer sheets are useless, and it is difficult for teachers and students to share answers during the exam. By 2019, 39,379 out of 50,124 junior secondary schools implemented CBT. We conduct a difference-in-difference analysis, comparing the exam score trend of schools that switch to CBT with the exam score trend of the schools that still took the exam on paper in 2019.

We find that test scores decline dramatically after the introduction of CBT. School-level means declined by 5 points on a scale from 0 to 100 (0.4 standard deviations). Schools with response patterns that indicate cheating experience a steeper drop in their test scores of almost 10 points. The drop in scores for the other schools is not significantly different from 0. We also find that the effect is similar for schools that already had computers before the intervention and schools that got them during the intervention or used computers of other schools. This suggests that the drop in scores is not driven by a lack of computer skills. Because of this drop in exam scores, the correlation between school rankings based on exam scores before and after the introduction of CBT is low.

Test scores rebound within two years after introducing CBT, suggesting that barriers to cheating provide incentives for learning on high stakes exams. Since cheating is virtually impossible, an improvement in exam scores can only be achieved through students and teachers exerting more effort into learning outcomes.

Finally, we find evidence of spillover effects from CBT within districts. Cheating was concentrated in certain regions, pointing to the existence of a local cheating culture in those areas. We find that if more schools in a district make the switch to CBT, cheating in schools that have not yet switched also declines more, suggesting that CBT not only eliminates cheating but makes it less socially permissible. As our findings also suggest that learning outcomes improve when cheating is no longer possible, we conclude that CBT helped Indonesia move from a cheating culture to a learning culture.

Authors

• Emilie Berkhout, University of Amsterdam

• Menno Pradhan, Vrije Universiteit Amsterdam