Session Submission Summary

From where you are to where you want to be: How information shapes goals, behaviours and motivation of actors in education

Tue, April 27, 6:15 to 7:45am PDT (6:15 to 7:45am PDT), Zoom Room, 133

Group Submission Type: Formal Panel Session

Proposal

Countries need regular, reliable and relevant information on the performance of education systems to promote mutual accountability for actors in the system. Good data is crucial for identifying where countries currently are on established goals, for articulating desired and realistic targets, and for tracking progress on these targets and goals. Education systems in low- and middle-income countries need to produce credible information on learning and system performance if they are to combat the learning crisis and create systems aligned for learning. In short: Accountability is difficult to achieve without reliable information.

Nonetheless, perhaps counterintuitively, the process of data collection itself can prompt behaviors that undermine the goal of gathering reliable information. Distorted data artificially increases system performance and presents a shortcut for actors to skew information rather than actually improve outcomes. As this panel shows, this is the case for high-stakes settings of exams as well as low-stakes settings where actors do not expect direct rewards or ramifications. As a result, learning outcomes may be even lower than previously assumed, and new ways of collecting data on system performance are required. The panel also highlights multiple pathways to increase accountability and alignment for learning, presenting the positive impact of changing the type of information that is gathered as well as the data collection and dissemination processes that alter the perception, motivation and options of stakeholders.

The panel opens with two empirical papers on the magnitude and effects of widespread cheating, capturing cheating as a systemic rather than individual issue in education, skewing the information available for more effective policymaking and reform design. The first analysis titled ‘Myths of Official Measurement: Auditing and Improving Administrative Data in Developing Countries’ uses representative samples of two Indian states to measure the magnitude of manipulation in a low-stakes setting, analysing administrative census data on learning achievement. The author finds that official results are extensively overstated due to a combination of student copying and teacher assistance, but also shows that only some ways of administering assessments are prone to manipulation. In contrast to paper-based assessments, tablet-based assessments eliminate the possibility of cheating and the inflation of results. This technology-led solution has the potential to improve the reliability of learning outcomes data in India and beyond.

Such a solution is also at the center of the second paper, ‘From Cheating to Learning: An Evaluation of Fraud Prevention on National Exams in Indonesia’, which evaluates the impact of computer-based testing (CBT) on national exams. As CBT makes cheating difficult, there is a significant initial decline in test scores. However, test scores recover after two years, suggesting that teachers and students refocus classroom efforts on learning where before attention was on the ‘shortcut’ of cheating. Not least, detecting that cheating is concentrated in certain regions, the authors conclude that CBT enables shifts in local norms that cause transitions from a cheating culture to a learning culture, creating positive spillovers of CBT even to non-treated schools in the same district. This highlights the importance of the cultural, normative dynamics in shaping actors’ perception of and attitude towards information and the data collection process.

The third paper, ‘Assessing the Assessment: Taking Stock of Learning Outcomes Data in India’ presents results from a comparative study assessing the reliability of three nationally representative surveys on learning outcomes in India. It includes the Annual State of Education Report (ASER), which is administered by the ASER Centre, the National Achievement Survey (NAS), conducted by the central government, and the independently administered India Human Development Survey (IHDS). The authors find that NAS results are significantly and systematically higher than ASER and IHDS state averages and do not accurately reflect states’ relative performance. These mixed results on accuracy, reliability and comparability not only demonstrate the need for improved data quality, but also underscore the administrative and political purposes of inflating data on system performance.

The fourth paper, ‘Through thick and thin: Varieties of information and incentives in strengthening accountability for remote schools in Indonesia’ is a qualitative reanalysis on the KIAT Guru intervention that was implemented in Indonesia to increase teacher performance and accountability, and which significantly increased learning outcomes. The study focuses on the different types of accountability (‘accounting-based’ and ‘accounts-based’) that can be created through collecting and disseminating different types of information (‘thin’ and ‘thick’). The authors find that part of Kiat Guru’s success is due to embedding thin information such as camera-validated teacher attendance in thick, account-based information such as narrative discussions, shedding new light on the interplay of types of information to increase accountability in education systems.

The fifth study, ‘School Inspections, Bureaucrats, and School Quality: Experimental Evidence from Tanzania’ discusses the impact of the School Quality Assurance program. This nationwide reform, aimed at improving the school inspection process, shifted from a focus on punitive measures to an approach rooted in diagnostic recommendations and support for schools. This changed the type of information gathered during inspections, and allowed for a more holistic assessment of learning, teaching and management as well as more targeted recommendations shared with stakeholders. As a result, the reform not just changed what information was gathered and used, but how the process was communicated to create a shared understanding between stakeholders about how to achieve goals. These findings underscore the importance of perception and constructing data collection processes such that stakeholders feel supported rather than monitored and punished.

Overall, the panel highlights the need for thick, credible and relevant information to guide reform efforts, and emphasizes the importance of designing data collection processes carefully so as not to incentivize unintended behavior, antagonize stakeholders and undermine data quality.

Sub Unit

Chair

Individual Presentations