Rationale for School Gaming in Test-Based accountability systems: an analysis of perverse incentives in Brazil

• In Event: The Dissemination and Enactment of Global Educational Reforms in Brazil: unpacking the political economy and (undesired) impacts of test-based managerial accountability

Proposal

In test-based accountability systems, different actions to raise the test scores are adopted. We can refer to ‘teaching to the test’ and ‘narrowing the curriculum’, which, although not illicit, may have negative effects on the quality of teaching and instruction. In this research paper, instead, we focus on teachers’ illicit actions aimed at increasing the overall level of student performance, such as teacher cheating and the exclusion of students from tests. The rationale and motivations of opportunistic behaviors may be strongly related to the characteristics of the accountability and standardized assessment systems. When tests are framed in high-stakes accountability systems, with formal consequences for students, teachers or schools, dishonest behaviors may be encouraged to raise aggregated student scores (Stecher, 2002). For instance, some studies demonstrate that high-stakes tests tend to increase the likelihood of students and teachers cheating (Jacob & Levitt, 2003), or to encourage teachers to respond strategically to the incentives by placing certain students in special education (Cullen & Reback, 2006; Figlio & Getzler, 2002; Jacob, 2005).
As regards the method, we conducted logistic regression models with fixed effects to establish to what extent certain characteristics of federal test-based accountability systems in Brazil are associated with the probability of adopting opportunistic strategies to illicitly increase the test results. By opportunistic strategies we refer to the teacher cheating in the tests, while the characteristics refer to explicit pedagogical and financial incentives (high and low-stakes tests), which are provided unevenly along the Brazilian states. We identify teacher cheating with the method proposed by Jacob and Levitt (2003) to detect illicit behaviors adopted by teachers. This method combines two indicators. The first is the Unexpected Test Score Fluctuations (Jacob & Levitt, 2003), which refers to unexpected scores gains that can be explained by cheating. A classroom will be suspected of cheating if unexpectedly large gains are followed by lower than usual test score gains for the same students the following year. The second is a composite indicator of Suspicious Answer Strings, which is a combination of four measures (Jacob & Levitt, 2003): the unlikely block of identical answers given to consecutive questions in the classroom, the classroom average variance across all test items, the variance in the degree of correlation across questions within a classroom, and the extent to which a student’s response pattern differs from other students with the same aggregate score that year.
This paper significantly contributes to the policy debate, providing empirical evidence to understand how strategical illicit behaviors are adopted in systems with extrinsic incentives based on the standardized tests’ results. The results allow us to confirm that teachers are more likely to game the system through cheating when a test has formal consequences for them. The strategic responses of teachers are sensitive to variations in the social composition of the school and are correlated with other illicit strategies which occur in parallel, such as the exclusion of students from the tests. Relevant policy implications for the Brazilian test-based accountability system are discussed.

Authors

• Andreu Termes, Universitat Autònoma de Barcelona

• Gerard Ferrer-Esteban, Autonomous University of Barcelona

• Antonina Levatino, Universitat Autònoma de Barcelona