Using ChatGPT for trans-adaptation of a Life-Skills Assessment in Kiswahili

• In Event: The uses of artificial intelligence (AI) in educational settings

Proposal

Using ChatGPT for trans-adaptation of a Life-Skills Assessment in Kiswahili

Authors: Dhiraj Anand, Fernanda Gandara & Jacktan Magesa

Introduction: Using assessments across different geographies and populations than where they were originally developed is a common practice. There is abundant literature on the translation and adaptation of tests developed in one language, typically English, for use in different languages and populations (Hambleton et al., 2005; Sireci & Allalouf, 2003;). Merely translating a test may overlook deeper issues of equivalence. Consequently, translation alone is insufficient; it is merely the first step in the broader process of test adaptation (ITC, 2017). Instead, translation needs to be thought as the initial stage of adaptation, a process that must account for cultural, idiomatic, linguistic, and contextual aspects (Hambleton et al., 2005).
Since 1992, the International Test Commission (ITC) has proposed guidelines for cross-cultural translation and adaptation of psychological instruments (ITC, 2005). The most recent ITC guidelines (2017) extended their scope to include all activities involved in test adaptation, such as evaluating construct equivalence and conducting validity studies, to name a few. The framework provided by the ITC guidelines emphasizes three key principles: construct, method, and item equivalence (ITC, 2017). Construct equivalence ensures that the test measures the same psychological construct in the new context. Method equivalence involves ensuring that the procedures and formats used in the test administration are comparable across different contexts. Item equivalence ensures that individual test items function similarly across different cultures and languages. These principles are crucial for issues pertaining validity of the adapted test scores and intended uses. The guidelines highlight the importance of cultural sensitivity, urging researchers to consider cultural differences that affect how test items are interpreted and responded to. The ITC also advocates for rigorous methodological approaches such as conducting empirical studies to verify equivalence (ITC, 2017).
Despite these comprehensive guidelines, the process of translation and adaptation remains resource-intensive and time-consuming. It involves multiple stages of review to ensure that the adapted test maintains its integrity. In recent years, the development of Large Language Models (LLMs) and Artificial Intelligence (AI) tools, such as ChatGPT, Gemini, and Llama, has opened new possibilities for translations (Huang & Liu, 2024: Kunst & Bierwiaczonek, 2023; Siu, 2023). Studies suggest that machine translations from English to other European languages, similar in vocabulary, grammatical structure, and cultural content, achieve the highest statistical convergence with human translations. However, for dissimilar target and source languages, human involvement remains essential (Kunst & Bierwiaczonek, 2023). The literature has highlighted the potential and limitations of AI tools for translating and adapting assessments, and call for further research, experiments, and evidence to better understand these tools' capabilities and limitations.
This study is designed to assess ChatGPT's proficiency in trans-adapting (translating and adapting) a life skills assessment tool between English and Kiswahili in an applied setting. Specifically, it will assess the extent to which ChatGPT's trans-adaptations align with that of a team of experts. By comparing ChatGPT's and human outputs, the study explores linguistic accuracy, contextual relevance, and efficiency.
The study aims to answer three key questions:
1. How equivalent are ChatGPT’s trans-adaptations to those developed by a team of experts?
2. Which prompting styles yield better results in ChatGPT’s trans-adaptation?
3. To what extent can time and resources be optimized using ChatGPT?
Methodology: To respond to these questions, ChatGPT will be used simultaneously with the standard process that our organization follows for developing their life skills tools The translation and adaptation process taking approximately two to three months.
Typical steps in the translation and back translation process include:
• Sharing the English version of items with the country team’s translation team.
• Reviewing the local translation by content and language experts for cultural and contextual appropriateness.
• Sharing the final translation with the back translation team.
• Reviewing and discussing the back translation with the team for further adjustments.
These processes involve teams of experts in different parts of the world. Coordination among various teams during the translation process often leads to delays due to the availability of personnel and conflicting priorities.
In this study, we will use ChatGPT to translate approximately 50 items using various prompting techniques, including zero-shot prompting with context, role-based and specific instructions, few-shot prompting, and interactive prompting. We will adjust tone, sensitivity, and formality during the process. These techniques will be conducted alongside our team's translation process. ChatGPT's outputs will then be reviewed for equivalence by two human experts in the local language and subject matter. Furthermore, ChatGPT's Kiswahili translations of the items will be back translated to English by both ChatGPT and our team members. Human reviewers will analyse these back-translations for accuracy and equivalence.
Our findings will inform the potential integration of AI into global research activities and contribute to the evidence base for using AI in the trans-adaptation of assessment measures. Ultimately, this study aims to provide insights into the efficiency and effectiveness of AI tools like ChatGPT in enhancing the trans-adaptation process while maintaining high psychometric standards.

Authors

• Dhiraj Anand, Room to Read

• Fernanda Gándara, Room to Read

• Jacktan Magesa, Room to Read