Myths of official measurement: Auditing and improving administrative data in developing countries

• In Event: From where you are to where you want to be: How information shapes goals, behaviours and motivation of actors in education

Proposal

A central, yet understudied, component of state capacity is the quality of administrative data, which is a key enabler of effective policy making. For example, when delivering public services, governments need unit-level information to target interventions to individual persons, firms, schools or hospitals. I study administrative data integrity in the context of education systems in India.

Using direct audit evidence, I show that levels of student achievement in a large Indian state, from a census covering approximately 7 million students annually, are severely inflated the proportion of correct responses to the same multiple-choice questions is, on average, 38.9 percentage points (pp) higher in Math and 33.8 pp higher in Hindi in the official test, from a base of 25.1% and 37.9% correct responses in the retest in the two subjects respectively, i.e. a doubling of reported achievement. This discrepancy between the proportion of correct responses across the two assessments is present across the full distribution of prior achievement (as measured the previous school year), although it is substantially higher for weaker students. However, even the distorted metric retains some information about the ordinal ranking of schools and students. This discrepancy in achievement levels reflects a combination of student copying and teacher-assisted manipulation of scores. Compared to other classes in the same school, the magnitude of this discrepancy is reduced by one-half in Mathematics to three-quarters in Hindi in middle school classrooms where, to inhibit student copying and grade inflation, the test design mandated multiple booklets and external grading. Direct observation of testing in some schools and qualitative interviews with teachers and education officials also confirm cheating in this setting.

In a follow-up randomized experiment, covering more than 2400 schools in a different state, I evaluate whether digital testing using electronic tablets could reduce distortion. I find paper-based assessments proctored by teachers severely exaggerate achievement, in both private and government schools. An analysis of suspicious responses flags 38-43% of classrooms in the paper-based testing arm for cheating. An independently-proctored retest in a random sample finds that students score 16-20 percentage points higher, on average, in the teacher-administered tests than in the retest. The magnitude of this distortion is comparable to the analogous sample from M.P. in Grade 8 (with multiple booklets and external grading) and indicates the generality of concerns around distortion even in the best-case scenario for typical government assessments. This distortion is substantially reduced in tablet-based testing. In contrast to paper-based tests, only 2-5% of tablet-based test classrooms are flagged as potentially cheating. The difference between the official assessment and the retest is much smaller in the tablet-test arm, and not statistically distinguishable from zero in most cases. These results suggest that administrative data may be compromised even without high-powered incentives. However, well-designed technology-aided interventions may facilitate improving data integrity at scale.

Author

• Abhijeet Singh, Stockholm School of Economics