Validating a Chinese Version of a Critical Thinking Assessment

• In Event: Valid Assessment of Learning Outcomes in Higher Education: Cross-National Comparisons

Abstract

As Chinese higher education undergoes rapid expansion, quality control has become a central concern for policymakers and educators in China. In 2002, with the aim to improve the quality of higher education, Chinese Ministry of Education initiated an effort, called the Quality Assessment of Undergraduate Education (QAUE), to evaluate the institutional effectiveness. However, due to the lack of transparency in the tools used to measure student learning outcomes (SLOs), QAUE has been criticized by scholars as failing to improve higher education quality (Author, 2013; Zhou, 2010). Given the urgent need for a standardized tool to provide direct evidence of SLOs, we reported on an effort to translate, adapt, and validate the Critical Thinking test from the ETS Proficiency Profile (EPP), an SLO assessment being widely used for accreditation, accountability, and institutional internal improvement in the U.S. (Author, 2011). Specifically, the purposes of the presentation are: (a) to evaluate the psychometric qualities of the critical thinking test in terms of internal structure, item-test correlation, and reliability; (b) to examine the relationship between this test and relevant assessments; and (c) to explore the performance difference across subgroups.
The EPP Critical Thinking test was translated into Chinese through the following steps: (1) two groups of experienced translators translated the test separately; (2) they then discussed the discrepancies until an agreement was reached; (3) a panel of assessment experts who were bilingual in English and Chinese reviewed and refined the translation; (4) a user acceptance study was conducted in China for additional translation adjustment and also to determine the proper testing time; (5) a final version of the Chinese EPP Critical Thinking test was created. To ensure the psychometric standards of this test, we pilot tested it among 2,023 college students from two institutions (i.e., an elite and a non-elite) in China. The following analyses were conducted to support the valid use of this test.
First, the dimensionality analysis revealed that the Chinese EPP Critical Thinking test consists of one general dimension of critical thinking and three subdimensions corresponding to the three contexts of the items. Second, the test showed satisfactory reliability (i.e., alpha = 0.76) and reasonable correlations with the college admission test in China (i.e., Gaokao; and a standardized English proficiency test for college students (i.e., CET-4; r = 0.30). Third, all the items showed moderate correlation with the total score, except for one item which needs further revision in translation. In addition, the subgroup analyses suggested that students who were male, majored in a natural science discipline, or were from a higher SES background significantly outperformed their peers .
In summary, the results suggested satisfactory psychometric quality of the Chinese EPP Critical Thinking test, which supports the use of this test in China’s higher education institutions. The approach we used to translate and validate this test could also benefit test developers who are interested in translating SLO assessments or institutions who are considering adapting existing SLO measures.

Authors

• Ou Lydia Liu, ETS

• Liyang Mao, IXL Learning

• Zhen Wang, ETS