Investigating the Factor Structure of the Personal Potential Index Using a Multilevel Factor Analysis Approach

• In Event: Problems With Interpretations of Multilevel Data: Extending Research Beyond Hierarchical Linear Modeling

Abstract

Finding ways to enhance candidate selection mechanisms have been a central ongoing concern in higher education. This interest has generated a host of research and has led to the development of measures to accurately assess the skills of individuals. Test development has primarily focused on cognitive measures such as the GRE® revised General Test. More recently, the attention has turned to developing personality (e.g., organization and teamwork) measures, which have been determined in previous research to be key determinants in graduate school success. Interest in such measures has greatly increased as they may augment the predictive validity of cognitive measures and narrow the gender and racial/ethnic gaps in admissions rates (Schmitt et al., 2009; Walpole, Burton, Kanyi, & Jackenthal, 2002).
Identifying effective ways to measure personality factors has been challenging. Typically, self-ratings have been used for this purpose; however, they have been criticized for being easily “faked.” Alternatively, ratings by others have been proposed to increase objectivity and fairness in measurement (Kyllonen, 2008). A key concern has been to minimize potential biases of raters (e.g., halo and ceiling effects introduced by rating candidates too highly, and differences in rating leniency). As raters’ effects nested with applicants may conflate the measurement of personality factors, a data analytic model enabling the analyses of variance at the applicant and the rater levels may be needed to obtain accurate estimates of personality profiles of individuals.
In this study, we demonstrate the use of a multilevel factor analysis approach applied to others’ ratings. We use data from the ETS® Personal Potential Index (PPI) developed by Educational Testing Service (ETS®). The PPI is the first large-scale assessment allowing for a quantifiable evaluation based on others’ ratings of personality attributes of graduate applicants (i.e., Knowledge and Creativity, Communication Skills, Teamwork, Resilience, Planning and Organization, and Ethics and Integrity), which have been deemed critical to graduate school success. Such attributes are not currently captured by standardized cognitive tests for graduate school admissions (Kyllonen, 2008). We used data from a sample of 19,843 applicants applying to various graduate programs across the United States. We investigated the factor structure of the PPI using a Confirmatory Factor Analysis (CFA) model based on two approaches. A traditional CFA model, which ignores the effects of raters nested with applicants; and a second (multilevel CFA) approach which takes data nesting into account.
Findings revealed that the two approaches yielded different results. The single-level CFA provided support for the intended six factor structure of the PPI. Multilevel CFA results provided support for a different structure: two factors at the between (applicant) and four factors at the within (rater) levels. Possible reasons for such discrepant results will be presented and implications for measures in which data nesting occurs discussed. This is the first study that uses a multilevel CFA to the analyses of others’ ratings thus advancing the field of noncognitive measurement by addressing a key challenge: separating applicants’ from raters’ variance.

Authors

• Maria Elena Oliveri, Educational Testing Service

• Steven L. Holtzman, ETS

• Chelsea Ezzo, ETS