Chatting up Success? A Quantitative Study on English as a Second Language (ESL) International Students’ Use of ChatGPT in Academic Writing

• In Event: Student agency and adaptation: artificial intelligence, social media, and diverse learning spaces

Proposal

Introduction
In recent years, artificial intelligence (AI) has advanced significantly, with OpenAI's ChatGPT, launched in November 2022, exemplifying this progress. The impact of ChatGPT on higher education is profound, prompting discussions on its integration into curriculum design, assessment, and concerns about ethics and plagiarism (Kim et al., 2023; Hong, 2023; Johnson, 2019). Research also highlights its ability to assist in generating outlines, summarizing articles, and analyzing research papers (Thorp, 2023; Poldrack et al., 2017). However, concerns remain about the depth, word count, referencing errors, and plagiarism in ChatGPT-generated texts (Kumar, 2023; Barrot, 2023). Despite these advancements, limited research has focused on how English as a Second Language (ESL) international students use ChatGPT for academic writing. Academic writing has long been a challenge for ESL students, with difficulties in grammar, citation, plagiarism, and unfamiliarity with English writing styles (Perry, 2016; Phakiti et al., 2013). Having been used as a writing tool and writing assistant, ChatGPT may offer a valuable tool to help overcome these barriers (Imran & Almusharraf, 2023).
This study explores ESL international students’ self-efficacy in academic writing and their use of ChatGPT. Drawing on Bandura’s (1984) social cognitive theory of self-efficacy, which reflects an individual's belief in their skills and capabilities related to English academic writing, we examined how these students perceive the influence of ChatGPT on their writing abilities. Our study aims to address the following question: How does the use of ChatGPT influence academic writing self-efficacy among ESL international students?
Methodology
Data was collected via an anonymous survey adopted from previous studies to investigate ESL international students’ habits and self-efficacy in academic writing using ChatGPT. The survey included three sections: demographics and ChatGPT user experience, perceived usefulness and ease of use based on Davis’ (1987) Technology Acceptance Model (TAM), and academic writing self-efficacy (SE) (Bruning et al., 2013). Independent variables measured were TAM variables—perceived usefulness (PU), attitude (ATT), and behavior intention/intention to use (BI). Dependent variables focused on writing self-efficacy, including writing essentials (WE), relational reflective writing (RRW), and creative identity (CI).
Data Analysis
Descriptive statistics was conducted to explore students’ user experience and overall perceptions of using ChatGPT for academic writing. Multivariate analysis of variance (MANOVA) was conducted to investigate how user experience influences their perceptions of using ChatGPT. Lastly, Canonical Correlation was used to analyze the relationship between TAM and students’ academic writing SE, focusing on how TAM impacts their academic writing SE.

Participants
A total of 327 survey responses were collected, with 246 from ESL international students. Of the respondents, 40.7% were female and 57.3% were male. Among them, 82.5% of the participants were undergraduate students, 16.3% were graduate students, and 1.6% were professional or certificate students. Regarding disciplines, 28% were in STEM, 21.1% were in Arts and Humanities, 25.6% were in Social Science, 25.6% were in Medicine, and 8.1%were in Business. The sample represented 58 countries, with 47.2% from Asia, followed by Europe (17.9%), Africa (9.3%), Oceania (6%), and both South and North America (2.8% each).
Results
Frequency of Using ChatGPT
The survey revealed that 62.2% of the participants use ChatGPT in academic writing about half the time or less, 28% use it most of the time, and 11.8% use it all the time.
​​To explore the impact of ChatGPT usage on students' TAM and academic writing SE, a one-way MANOVA was conducted based on different frequencies of ChatGPT use. Box’s M test showed unequal covariance matrices among the groups (F = 2.71, p < .001), Pillai’s Trace was then used. Pillai’s Trace revealed significant differences in TAM and Academic writing SE among students with varying ChatGPT usage, with a large effect size (F(6, 236) = 5.93, p < .001, η² = .13). Specifically, students’ PU varied significantly (F(3, 245) = 32.31, p < .001), as did their ATT (F(3, 245) = 30.72, p < .001) and BI (F(3, 245) = 38.64, p < .001). In terms of academic writing SE, significant differences were found in WE (F(3, 245) = 30.14, p < .001), RRW (F(3, 245) = 31.07, p < .001), and CI (F(3, 245) = 23.90, p < .001). These results show that students who consistently use ChatGPT scored highest across all categories compared to other groups.
Attitude towards Using ChatGPT
Table 1 shows that international ESL students generally have a positive TAM toward ChatGPT and higher academic writing SE. A Canonical Correlation Analysis (see Figure 1) tested the relationship between TAM and academic writing SE, with results indicating a strong positive association (rc = 0.89, Λ = 0.21, F(9, 581) = 58.77, p < .001). Using a cutoff of 0.30 (Tabachnick & Fidell, 2001), PU (.98), ATT (.93), and BI (.93) from TAM correlated with academic writing SE, while WE (.91), RRW (.99), and CI (.89) from Academic SE correlated with TAM. PU explained 96% of the variance, ATT and BI 86%, RRW 98%, WE 83%, and CI 79%. A strong positive correlation (rc = 0.89) was found between TAM and academic writing SE. PU was the most significant variate (.98), with a one standard deviation increase in PU leading to a 0.59 standard deviation rise in academic writing SE. ATT and BI also contributed as well, suggesting that international ESL students with high TAM, especially strong PU, ATT, and BI, tend to have higher academic writing SE, particularly in WE and RRW, but lower CI.
Conclusions
This study highlights the significant relationship between international ESL students’ use of ChatGPT and their academic writing SE. The results suggest that students who frequently use ChatGPT report higher levels of PU, ATT, and BI, which in turn positively correlates with their academic writing SE, particularly in areas such as WE and RRW. Given the growing prevalence of AI in higher education, it is essential for future research to continue exploring how tools like ChatGPT can be effectively integrated into academic settings to support ESL international students while addressing ethical and pedagogical concerns.

Authors

• Sanfeng Miao, Michigan State University

• Xi Lin, East Carolina University