Challenges Facing Translating International Comparative Assessment Instruments: The Case of isiXhosa in South Africa

• In Event: The Challenges of Assessing and Comparing Young Children in Different Cultures

Abstract

The aim of this paper is to describe the challenges experienced within an international comparative assessment project with regard to the translation of Grade 1 international assessment instruments into a non-standardised South African indigenous language and how these were addressed. The paper addresses the question: how can an international assessment best be adapted into a language for which there is no standardised form and still be comparable?

To optimise the validity of the translation, a combination of content analysis and statistical item analyses were used. Firstly, the contextualised South African English instrument was sent for translation into isiXhosa. After two translations, all translations were back translated into English (Martin and Kelly, 1996). The back translations were studied for inconsistencies against the original text. This was followed by a small scale trial in which the instrument was administered to a sample of 75 Grade 1 learners (25 per language group: Afrikaans, English and isiXhosa) from three schools. The assessment results were statistically analysed, to check for Differential Test Functioning (DTF) and Differential Item Functioning (DIF) (Bond and Fox, 2015; Boone et al, 2014; Price and Oshima, 1998). The dichotomous Rasch Model was applied using the Winsteps program (Linacre, 2011). After the Rasch model was applied, the item measures were imported into SPSS v.23 and analysed for DTF for all three language groups. Adjustments were then made to the instruments.

Next, a larger scale field trial was conducted with 300 learners (100 per language group) who were randomly sampled from within 12 schools to further examine the instrument’s functioning. Evaluation reports were completed by the fieldworkers to identify any specific translation challenges they experienced in the field. Finally, a verification workshop was held with translators, fieldworkers and assessment specialists to adjust the final isiXhosa version of the instrument in such a way that all parties agreed would be most appropriate for Grade 1 learners in the Western Cape Province.

The statistical analysis of the literacy items revealed that the test as a whole, functioned in a similar manner across all three language groups. When examining the numeracy section, the test also did not function differently for the isiXhosa group when compared to the English and Afrikaans groups. The DIF analysis revealed that only three out of the 91 literacy items displayed DIF for the isiXhosa group, while none of the numeracy items showed any differential functioning. These results provided support for the adaptations and the functioning of the instruments overall. The feedback reports from the fieldworkers during the second round of fieldwork, as well as the verification workshops were instrumental in the refinement of the assessment instrument and reaching consensus.

The paper concludes that a combination of approaches made it possible to translate and verify the translation and adaptation for a language with a great deal of variation across distance and within culture.

This is important because it shows that translation/adaptation into languages with many dialects is greatly challenging, but solutions are possible and some are offered in this paper.

Authors

• Sarah Jane Howie, University of Pretoria

• Celeste Combrinck, University of Pretoria, South Africa

• Peter B Tymms, Durham University