The Predictive Validity of Noncognitive Skills for Graduate and Professional Student Success: Some Initial Findings

• In Event: 35.080 - Validity and Fairness Issues in Assessing Higher Education Students

Abstract

Given the breadth of individual characteristics that determine successful graduate and professional student performance, experts in graduate education have advised that, in addition to standardized tests of cognitive skills, valid assessments of non-cognitive skills also be used in graduate and professional school admissions (Briel et al., 2000; Enright & Gitomer, 1989; Kuncel et al., 2001; Kyllonen et al., 2005). This guidance is opportune given mounting pressures for greater efficiency in higher educational outcomes (see, e.g., Olson, 2010) and findings such as less-than-optimal degree progress for doctoral students in general (Sowell et al., 2008). We present findings from the first predictive validity study of the ETS Personal Potential Index (PPI), an online and standardized evaluation by multiple third parties of the following skills of applicants to graduate and professional school: Knowledge and Creativity, Communication Skills, Teamwork, Resilience, Planning and Organization, and Ethics and Integrity (Kyllonen, 2008).

With PPI dimensions individually and composites serving as predictor variables, the outcomes criteria in this study included first-semester and first-year graduate grade point averages, student standing (e.g., good standing versus academic probation), measures from surveys to be administered to faculty and students who know graduate school enrollees, and measures from surveys to be administered to enrollees themselves (i.e., self-report measures). The surveys assess respondents’ perceptions of enrollees’ degree progress, engagement in relevant professional development (internships, teaching assistantships, publishing, etc.), demonstration of program-relevant non-cognitive skills (e.g., resilience), and engagement in prosocial and counterproductive behaviors (e.g., helping other students). There is substantial research support for the importance of these outcomes criteria in a broad criterion space and the ability of non-cognitive measures to predict them (see Camara et al., 2003; Credé & Kuncel, 2008; Kuncel et al., 2001; Oswald et al., 2004; Robbins et al., 2004; Schmitt et al., 2003, 2009; Schwager, et al., under review).

We examined PPI validity directly as well as the extent to which it provides incremental validity above undergraduate GPA and GRE scores. Analyses include correlation, regression, and group-difference comparison methods. Some analyses will be multi-level; others will use sample-size weighted averages across groups. Analyses are in the aggregate and then broken down by broad discipline area and degree level. For example, in Table 2, we reported initial observed validity coefficients (zero-order correlations) for STEM programs based on sample-size weighted averages. The STEM programs came from six universities and have the largest sample sizes of the main disciplines.
Our analyses come from a sample consisting of 386 students enrolled in one of six U.S. graduate and professional institutions (the first of three cohorts for whom we are collecting longitudinal data). We use data from the group of 829 admitted individuals and 1,455 applicants from which the enrollees come in order to adjust validity coefficients for univariate range variation (Hunter & Schmidt, 2004). We also assessed the reliability of predictors and criteria. Moreover, we will ascertain the generalizability of our results (i.e., whether validity coefficients are subject to moderators and thus should be reported for subgroups) using procedures of Hunter and Schmidt (2004).

Authors

• David Marc Klieger, Educational Testing Service

• Chelsea Ezzo, ETS

• Jennifer Bochenek, ETS

• Frederick A. Cline, ETS