Judges Versus Juries: Who Should Provide Gold Standard Labels in the Era of LLMs?

• In Event: Classification and LLMs

Brief Overview

We consider the problem of learning a set of ground truth labels using an LLM (judge) and human annotators (a jury). We formally show an interpolation-extrapolation trade-off: LLMs are more reliable for problems commonly observed in training data.

Authors

• Adam Austin Bouyamourn, Princeton University

• Arthur P Spirling, Princeton University