Decoding the Curriculum Beyond the Anglosphere: A Methodological Framework for Non-English Textbook Analysis

• In Event: Teachers, Pedagogy, and Professional Practice in the Age of ICT

Proposal

The evolution of Natural Language Processing (NLP) has enabled large-scale investigations into the representation of marginalized groups in educational materials, revealing persistent underrepresentation and stereotyped portrayals (Crawfurd et al., 2024; Lucy et al., 2020). However, significant technical challenges remain, particularly in non-Anglophone contexts where NLP resources are scarce. Controversies persist regarding the extent to which powerful pre-trained models like BERT perpetuate or even amplify existing societal biases, creating a risk of reinforcing inequalities through automated analysis (Taeb et al., 2022; Chang et al., 2019). This study addresses this methodological challenge directly. It asks: How do different NLP pipeline architectures and language-specific models systematically affect the measurement of gender and racial representation in educational texts? By conducting a comparative analysis of six distinct NLP pipelines (including BERT and spaCy variants) on a corpus of 102 Brazilian humanities textbooks, this research aims to develop a robust methodological framework for analyzing representation in non-English materials. The goal is to provide a transparent, replicable protocol for researchers and policymakers seeking to understand curricular content focusing on equity and inclusion.

This study is positioned at the intersection of computational linguistics and critical education studies. While the core of the paper is methodological, its significance is framed by the understanding that textbooks are complex social artifacts where symbolic disputes are contested (Apple, 2013). Biased educational content, including the "hidden curriculum," can reinforce harmful stereotypes and shape student perceptions of national identity and societal norms (Jackson, 2002; Schissler, 2009). The choice to focus on gender and race is, therefore, deliberate, as these are critical axes of inequality in education (Cerna et al., 2021). The theoretical underpinning of this work is that the technical choices made in NLP research are not neutral; they have direct implications for social justice. An inaccurate or biased pipeline can obscure or even exacerbate the very inequalities it is designed to detect. Therefore, this research is guided by a commitment to methodological transparency and contextual adaptation, arguing that equitable tools are a prerequisite for equitable analysis.

The study conducted a comparative performance analysis of six representative NLP pipelines for Named Entity Recognition (NER) on a corpus of 102 Brazilian high school humanities textbooks from the National Textbook Program (PNLD). The pipelines were selected to test the effects of model architecture (BERT/Transformer vs. spaCy), language specificity (English, Multilingual, Portuguese), and the inclusion of coreference resolution. The analysis proceeded in two stages: first, text was extracted and cleaned from all 102 textbooks. Second, each of the six pipelines was used to perform NER, with entities mapped to demographic macro-categories for gender (male/female) and race/ethnicity (black, white, and other races and ethnicities). The performance of each pipeline was then compared based on two key metrics: coverage (the absolute number of relevant entities detected) and representational profile (the proportional distribution of detected entities across demographic categories).

The results demonstrate that technical choices in pipeline design produce systematic and significant effects on the measurement of representation. For gender, all pipelines identified a persistent asymmetry in favor of men. However, both the overall number of mentions (coverage) and the specific gender ratio varied widely. The spaCy 3.8 (Portuguese) pipeline achieved the highest coverage by a large margin, identifying over 90,000 mentions of people. In contrast, BERT-based pipelines identified fewer than 4,000. Interestingly, the Multilingual spaCy pipeline, while having lower coverage, identified the highest relative proportion of women (28.19%), suggesting greater sensitivity to female-marked cues. For race and ethnicity, the differences were even more stark. The spaCy 3.8 (Portuguese) pipeline was the only one to capture a diversified distribution that reflected the Brazilian context, identifying large numbers of "afro-brazilian" (36.02%) and "other-minority" (38.12%) mentions. In contrast, all BERT-based and English-native spaCy pipelines produced highly concentrated racial profiles, with "white" entities accounting for 68-70% of mentions and almost completely failing to detect "afro-brazilian" entities. This suggests a critical dependency on language-specific and culturally-attuned taxonomies.

This study's critical methodological contribution is its empirical demonstration of the trade-off between coverage and specificity in NLP-based textbook analysis. For large-scale screening of non-English corpora, language-specific models like spaCy 3.8 for Portuguese prove to be superior candidates, as they combine high coverage with greater sensitivity to locally relevant demographic categories, whereas more generic models can produce misleadingly narrow profiles. The primary significance of this research is the development of a set of practical recommendations for conducting this work responsibly. Future research will build on this framework to establish a general pipeline for assessing and mitigating "contextual loss"—the nuances lost when applying generic tools to specific languages. This involves creating "gold-standard" hand-coded datasets for non-English languages to serve as benchmarks, providing a cost-benefit analysis to help researchers and practitioners weigh the trade-offs between using generic versus context-specific tools. By providing a transparent, evidence-based framework, this broader research agenda offers a promising path for researchers and policymakers to develop more reliable and equitable tools for auditing curricular materials worldwide.

Authors

• Alan Pedro da Silva, Blavatnik School of Government, University of Oxford

• Filipe Recch, The University Of Pittsburgh School of Education

• Felipe José de Queiroz Sarmento, Center for Excellence in Social Technologies

• Diego Dermeval Medeiros da Cunha Matos, Federal University of Alagoas

• Bruno Pimentel