“You cannot sound like GPT": How multilingual scholars navigate cutting edge language ideologies in academic publishing

• In Event: Language and AI Pedagogies, Technologies, and Practices

Proposal

Indexed research papers are published almost entirely in English, limiting the participation of readers and writers from most of the world's population. ChatGPT has thus been celebrated as a tool for scholars whose first language is not English to instantly adapt their writing in preparation for English-language publication. Rather than examine ChatGPT as a solution, we theorize sites of ChatGPT adoption as indicators of linguistic exclusion. How do language-marginalized scholars use this tool for the purpose of navigating English hegemony in publishing? Is such use associated with a change in the expression of language ideologies by peer reviewers? Does the tool change how language-minoritized scientists' experience the challenges and reap the benefits of English hegemony in scientific publishing?

To answer these questions, we examine the case study of an influential, international computer science conference. Using computational methods, we analyze critiques and praise of writing “clarity” across the conference's almost 80,000 peer reviews of over 20,000 papers before and after the introduction of ChatGPT. We theorize reviewer feedback on writing not as measurements of language clarity but as an indicator of language ideologies about what kind of writing should be published. Yet these written critiques can only tell a partial story about the expression and experience of language ideologies in this scientific community. Using concurrent nested triangulation, we conduct semi-structured interviews with 13 language-marginalized authors from across five continents and one area chair who has been with the conference since its founding.

Our results indicate a shift in the way reviewers talk about the writing of authors from certain countries in the years following the release of ChatGPT, especially authors from Asia. However, responses from interviewees indicate that the effect of this tool at preventing language critique may be already be shifting in efficacy. Prior to the release of ChatGPT, reviewers might identify features of writing to infer that an author's first language is not English- our interviewees mentioned variations in subject-verb agreement or atypical syntax. After the release of ChatGPT, interviewees describe how certain words or flowery language they associate with the tool- despite syntactically and morphologically matching the norms of the dominant linguistic group- may now come to signify the authors' “non-native" identity. This innovative way of expressing and experiencing language ideologies then becomes a mechanism for continued social and cultural sorting into hierarchies of academic worth. In response, linguistically-marginalized authors describe using the tool to cover their language identity while simultaneously hiding the fact that they use the tool; as one author argued, "You cannot sound like GPT".


Our interviewees indicate the presence of a secondary process of social sorting into which the affordances of ChatGPT become embedded. Interviewees discuss how in the past, language “errors" in papers published English-language publications signaled to them that the science is not trustworthy. To these interviewees, cues that ChatGPT-generated text is present in writing also become signals of untrustworthy science- not necessarily because ChatGPT is used, but because the authors may be using the tool to cover up their identities as scientists who presumably cannot otherwise publish in English. These findings suggest that academic critiques of writing in peer review may be more closely tied to biases about country identity than to the functionality of mutual intelligibility.

With this work, we seek to understand the role of ChatGPT in the reproduction of inequality in higher education through ideologies which conflate producers of "good English” with producers of ``good science."

Author

• Haley Lepp, Stanford University