Where you place the stakes matters: exploring test-based accountability and participation in shadow education

• In Event: Mechanisms for Educational Change: Policy, Networks, and Accountability Structures

Proposal

Accountability in education remains an important consideration for policymakers. Since the move to the New Right in the UK and US in the 1980s there has been increased pressure on ensuring public investments are efficiently and effectively managed (Figlio & Loeb, 2011). Questions on who should be accountable for what in education, and how to hold these individuals to account, have expanded with accountability becoming a focal point in many countries (UNESCO, 2017). Student test scores have gained prominence as a seemingly objective measure of progress in education. Motivated by the ‘learning crisis’ (Benavot & Smith, 2020) and with their position reinforced by the ‘quality turn’ (Sayed & Moriarty, 2020) that corresponded with the movement from the Millennium Development Goals to the Sustainable Development Goals, some have argued that there is an “increasingly ‘common-sense’ notion that testing is synonymous with accountability, which is synonymous with education quality” (Smith, 2016, p. 7). Therefore, to improve education quality there are often generic recommendations to increase accountability, which will then be evaluated against change in test scores.

Missing from the push toward greater accountability is a more nuanced understanding of how different accountability approaches motivate behaviour and lead to desired (or undesired) outcomes. A general call for more accountability assumes that all approaches to accountability are beneficial at all times in all situations. However, accountability pressure can be placed on multiple different actors in education and understanding who responds to what pressure, and how, is important in designing accountability policy. Historically high-stakes exams have placed responsibility of testing scores squarely on the students. However, over the past 30 years there has been a transformation from traditional high stakes testing to testing for accountability, which places intentional or unintentional positive or negative consequences on educators (teachers and administrators) for their student’s performance (Smith, 2014). Understanding both student focused and educator focused accountability is important because different approaches to testing are situated on different philosophical models and likely to lead to diverse student outcomes (Harris & Herrington, 2006). Smith (2016) has recognized this in describing the Global Testing Culture. Defined as “a culture in which high-stakes standardized testing is accepted as a foundational practice in education and shapes how education is understood in society and used by its stakeholders” (Smith, 2016, p. 10), Smith describes how cultural scripts valuing test scores creates expectations for the behavior of educators, students, and parents, with non-compliance often shunned by peers and society.

This study contributes to the conversation of how test scores are used in accountability policies to shape behavior by exploring the differential impact of placing high stakes on students versus educators. Using participation in private tutoring or shadow education as an example, we create a unique dataset to capture the presence of student focused high stakes testing policies and educator centered testing for accountability policies. The result is a 2 x 2 grid that maps countries into four categories: no test-based accountability, educator only test based accountability, student only test based accountability, and educator and student test based accountability. While there is substantial evidence indicating increased prevalence of shadow education in countries that link high stakes to students (Bray, 2009; Bray et al., 2014; Buchmann et al., 2010; Entrich, 2018; Park et al., 2016), less attention has been given to how placing high stakes on educators influences rates of shadow education.

Using our four country level policy categories, we apply Hierarchical Linear Generalized Modelling to predict participation in shadow education. To increase the sample size for analysis a combined measure of shadow education is created drawing from 2018 PISA and 2019 TIMSS datasets. Initial results suggest that there is a clear distinction in participation in shadow education based on how student test scores are used in the country. When student test scores are used to hold students accountable by linking scores with student access to further education or different types of further education, students are more likely to participate in shadow education. This differs from educator only test based accountability or countries that have no test-based accountability, where shadow education participation is relatively less common. Specifically, our early results suggest that while approximately 42% and 38% of students living in student only and educator and student combined test-based accountability systems participate in shadow education, only 25% of those that live in educator only and 18% that live in systems without test-based accountability participate in shadow education. The results have clear implications for designing accountability policy in education and provide empirical evidence indicating that where you place the stakes for student test scores matters in motivating actor behavior.

Authors

• William C. Smith, University of Edinburgh

• Huan Jiang

• Xiao Xu, Bytedance