When Predicting Performance, Less of a Bad Thing Is Better Than More of a Good Thing

• In Event: Situational Judgment Tests: Assessing Workforce Readiness

Abstract

Standard practice in personnel selection is to search for individuals with the highest levels of a target trait (e.g., general intelligence) to maximize the likelihood of positive outcomes of interest, such as college or job performance. However, recent research using situational judgment testing (SJT) (Elliott, Stemler, Grigorenko, Sternberg & Hoffman, 2011) corroborates the work of Meehl (1996) by indicating that what is most predictive of successful performance is not so much the presence of high levels of a target trait, but rather lower levels or the absence of a different target trait (e.g., foolishness). Consequently, success in the workplace may generally have less to do with stellar performance and more to do with not making glaring errors.

This presentation reports four different benchmarking studies carried out in three different corporate contexts in India. The first involves the use of a custom designed and proprietary Managerial SJT (N=50) while the other three are of a Sales SJT (Total N=150). For each scenario, the respondent had to mark the best course of action (Best Answer) and worst course of action (Worst Answer) among four to five choices. Three different scoring schemes were explored and compared. Each scheme includes a “Best Answer Score”, “Worst Answer Score” and a “Total Score” (a sum of Best Answer and Worst Answer). In Scheme 1
(-1,0,1) a candidate is given 1 when s/he correctly marks best answer/worst answer for that question. In case s/he marks best answer as worst or worst answer as best, s/he is given -1. For all other cases, s/he is given zero. The total score can range between -2 to 2.
In Scheme 2
(-1,0) scoring essentially mean a candidate is given -1 when s/he marks best answer as worst or worst answer as best. For all other cases, s/he is given zero. The total score will range from -2 to 0. For Scheme 3
(0,1) a candidate is given 1 when s/he correctly marks best answer/worst answer for that question. For all other cases, s/he is given zero. The total score will range from 0 to 2.

Evidence reveals that the “Worst Answer Score” with (0,1) scheme and “Best Answer score” with (-1,0) scheme show much higher correlations than other scores do with job performance indicators. Both scoring schemes are dependent on just the correct worst answer and not the correct best answer. In fact, in some of scoring schemes, the scores dependent on correct best answer add no incremental validity to that already achieved.

We conclude that SJTs can be good predictors of workforce readiness. However, their power is derived from the fact that there is not usually one best answer to a dilemma, but there does tend to be a worst response. Further, candidates who are not able to tell the difference between the “landmine” response and responses that are excellent or even average responses will likely end up being worse performers. These results also provide a cross-cultural replication in a non-Western context of a trend reported by other researchers.

Authors

• Steven Stemler, Wesleyan University

• Varun Aggarwal

• Siddharth Nithyanand

• Nisha Bhatt