Developing and Validating a Next-Generation Critical Thinking Assessment for Higher Education

• In Event: Developing and Evaluating the Psychometric Quality of Next-Generation Student Learning Outcomes Assessments

Abstract

Theoretical Framework and Purpose

Critical thinking is an important skill for college graduates to become effective contributors in the global workforce (Casner-Lotto & Barrington, 2006; AAC&U, 2011). As a result, it has been identified as a core student learning outcome (SLO) by many higher education institutions (ETS, 2013). In a review of existing critical thinking assessments, Liu, Frankel, and Roohr (2014) suggested that a number of these assessments either fail to provide a clear definition of the critical thinking skills they measure or lack of the satisfactory psychometric quality and instructional relevance. These issues might affect the quality of actionable data provided to institutions, and result in inaccurate evaluation of the accreditation and accountability. To address these issues, we developed a next-generation assessment to measure college students’ critical thinking skills. The purposes of this presentation are: (a) to introduce the process of assessment development, (b) to discuss how we addressed the issues identified in existing assessments, and (c) to provide validity evidence to support suitability of using this assessment in higher education institutions.

Methods and Preliminary Results

The critical thinking assessment was developed based on a synthesis of existing frameworks, empirical research, as well as the needs for institutions. It intends to capture both the analytical and synthetic dimensions of critical thinking. The dimensions are clearly defined so that the operational definitions are concrete enough to be translated into an assessment (see Liu, Frankel, & Roohr, 2014). To ensure the psychometric standards, we pilot tested the critical thinking test among 3,036 college students from 28 institutions. The following analyses have been conducted to support the score inferences.

First, the confirmatory factor analyses suggested that the two-dimensional structure (i.e., analytical and synthetic) is supported by the data, but two dimensions are highly correlated (e.g., over 0.90). Second, the individual-level total scores showed satisfactory reliability (see Table 1) and reasonable correlation with GPA, SAT scores, and self-rated critical thinking skills. The institution-level total score, analytical subscore, and synthetic subscore are also reliable; however, due to the high subscore correlation, the institution-level subscores are distinctive enough to report only if the sample size for that institution is greater than 150 students. Third, in terms of the performance difference between freshmen and seniors, after controlling for SAT scores, seniors scored 6.17 points (on the percent correct score scale of 0-100) higher than freshmen. For students in social science majors, seniors scored 11.40 points higher than freshmen; however, no difference was observed for natural science majors. In addition, 76% students rated the assessment being on the right difficulty level, and 86% claimed they had enough time to finish this test.

Study Importance

In summary, the results suggested satisfactory psychometric quality of the critical thinking assessment, which supports the use of this assessment in higher education institutions. The approach we used to develop and validate the assessment could also benefit test developers who are interested in designing an SLO assessment as well as test users who use SLO assessment scores for program/institution evaluation.

Authors

• Liyang Mao, IXL Learning

• Jun Xu, Educational Testing Service

• Ou Lydia Liu, ETS