LLMs as Judges: Score Range Bias and the Path Forward

Large Language Models (LLMs) are increasingly deployed as judges across various applications. But their reliability in scoring, particularly without references, remains a significant hurdle. The problem? Score range bias. LLM judges are incredibly sensitive to pre-defined score ranges. This bias isn't just an isolated incident. Models from the same family often exhibit similar traits.

Score Range Sensitivity

When LLMs assign scores directly, they tend to skew based on their sensitivity to the assigned score range. What does this mean in practical terms? It means that the scores you get may say as much about the score range setup as about the quality being evaluated. This isn't just a quirk. It's a systemic issue that undermines the credibility of LLMs when used as evaluators.

Why should anyone care? Because the promise of LLMs relies on their ability to act as objective, reliable judges in tasks like summarization, content evaluation, and more. Without overcoming this bias, their potential remains just that, potential.

Contrastive Decoding: The Solution?

So, how do we tackle this bias? Enter contrastive decoding. This method has shown a relative improvement of up to 11.7% in Spearman correlation with human judgments across various score ranges. This isn't just a minor tweak. It's a step towards making LLMs more reliable evaluators.

But let's not get ahead of ourselves. While contrastive decoding is promising, it's no panacea. The reality is, LLMs need a reliable foundation to be truly effective as evaluators. It's not just about adjusting the window dressing but addressing the core architecture and training paradigms.

The Bigger Picture

Here's what the benchmarks actually show: while progress is being made, there's a long way to go. The architecture matters more than the parameter count. Until we address deeper architectural issues, these models will continue to face challenges in consistency and reliability.

So, what's next? If LLMs are to step up as reliable judges, the focus needs to shift. Strip away the marketing and focus on real improvements. Otherwise, we'll keep circling the same issues. Are we ready to make those hard choices?