Understanding Interpersonal and Teamwork Knowledge Measured by Situational Judgment Tests: Establishing a Nomological Network

• In Event: Situational Judgment Tests: Assessing Workforce Readiness

Abstract

Cronbach and Meehl’s seminal article (1955) argued that a construct must be defined within a nomological network of related constructs to be properly understood. Construct validation entails placing constructs within a theoretical framework, and according to this rationale, validity cannot be established without theory generation and testing. However, in assessment research, the approach has often been to worry about predictive validity first and construct validity later.

The majority of research on situational judgment tests (SJTs) is characterized by this tendency, as it has focused largely on predictive validity rather than construct validity. The construct validity of SJTs remains unclear, as previous studies (e.g., McDaniel, Morgeson, Finnegan, Campion, & Braverman, 2001) have produced conflicting results in regard to the construct(s) measured by SJTs. Understanding why SJTs predict behavior is an important area for research (Lievens, Peeters, & Schoelhaert, 2008). Consequently, the purpose of this study is to integrate theoretical frameworks to explore the constructs measured by two SJTs: one created to measure knowledge about effective interpersonal interaction and one created to measure knowledge about effective teamwork.

A considerable amount of research has revealed that SJT scores are often correlated with measures of other constructs, such as cognitive ability and personality (McDaniel et al., 2001). This has contributed to the ambiguity concerning what SJTs measure, as these results suggest SJTs can assess a variety of constructs rather than one unitary construct. However, Motowidlo and colleagues (Motowidlo, Hooper, & Jackson, 2006) proposed a theory to explain how SJTs may assess one construct yet be correlated with measures of other constructs. They argued that SJTs measure procedural knowledge. This knowledge is described as being procedural in nature because it entails the understanding of effective and ineffective approaches to handling situations that involve interaction with others. If SJTs measure knowledge, it is unsurprising that cognitive ability would be predictive of SJT scores, as more intelligent people acquire knowledge more quickly and thoroughly than those who are less intelligent (Motowidlo, 2003).

However, why might personality traits predict knowledge scores? Motowidlo and colleagues (2006) argued that through life experiences, people form implicit beliefs about the effectiveness of the expression of particular personality traits in certain situations. According to this theory, implicit trait policies (ITPs) form as underlying personality traits interact with exposure to situations that afford opportunities for learning. A naturally agreeable person, for instance, may come to believe that behaving agreeably is effective in most situations, and is particularly effective when dealing with supervisors. When this belief is congruent with the behaviors required for effective performance in a domain, this belief represents knowledge.

In the present study, we administered a cognitive ability assessment, an assessment of written communication, a personality measure, and two SJTs (one about interpersonal knowledge and one about teamwork) to approximately 3,600 participants recruited through Amazon Mechanical Turk (AMT). Convergent and divergent relationships between study variables will be reported, and implications for theory and practice will be discussed.

Authors

• Michelle Martin, Educational Testing Service

• Patrick Barnwell