Management Accounting Section Midyear Meeting

Rating Behavior in Multi-Rater Evaluation Systems: Mutual vs. One-sided Ratings and Supervisor Adjustments

• In Event: Performance Evaluation

Abstract

In this paper, we examine rating behavior in multi-rater performance evaluation systems. Specifically, we study mutual rating relationships, whereby two employees rate each other contemporaneously under a multi-rater system. We use a proprietary dataset from an online platform retailer to show that demographic similarity (e.g., similar tenure or same nationality) and organizational proximity (i.e., same supervisor or same department) are positively associated with the likelihood of a mutual rating relationship. In addition, we show that mutual ratings are higher on average, with the evidence suggesting this is at least partly the result of reciprocal “uprating” behavior. Supervisors appear to be cognizant of the potential for bias in mutual ratings, as we find that they place relatively less (implicit) weight on mutual ratings vis-à-vis one-sided ratings when arriving at their overall assessment of an employee. Moreover, rating requests are more likely to be rejected by supervisors before the actual rating process if they would result in a mutual rating relationship, though this is less pronounced in the presence of organizational proximity between employees.

Authors

• Martin Artz, University of Münster

• Carolyn Victoria Deller, The Wharton School, University of Pennsylvania

• Sinja Leonelli, University of Chicago