Cutting Through the Noise: Improving AI Judgments with New Aggregation Method

Large Language Models (LLMs) are increasingly employed as judges in AI systems, but their single-sample judgments often lack precision. Conventional aggregation methods, like majority voting, fall short when ties emerge. A recent study tackles this issue by proposing an innovative aggregation scheme designed to reduce noise and improve accuracy.

The Problem with Current Methods

LLMs used for evaluating pairwise preferences are inherently noisy. Their judgments at the single-sample level can be inconsistent, especially when ties are allowed. Commonly used aggregation rules, such as majority vote and soft self-consistency, fail to adequately resolve these ties, resulting in unreliable outcomes.

The paper, published in Japanese, reveals a novel approach that could change the game. The researchers focus on inference-time compute (ITC) for evaluators, generating multiple independent samples per item to refine the evaluation process.

Introducing a New Aggregation Scheme

The heart of this new method is a distribution-calibrated aggregation scheme. It models three-way preferences using a Bradley-Terry-Davidson formulation on rating counts. This approach leverages both polarity, the margin among non-ties, and decisiveness, the non-tie rate, allowing it to distinguish between narrow margins and strong consensus. Compare these numbers side by side, and the improvement is clear.

What the English-language press missed: the benchmark results speak for themselves. Across various evaluation benchmarks, this method consistently reduces mean absolute error (MAE) and increases pairwise accuracy compared to standard baselines. When evaluated against human-consensus meta-labels, it matches or even exceeds the accuracy of individual human raters.

Why It Matters

This advancement isn't just a technical footnote. If AI systems can consistently outperform human evaluators, it raises a provocative question: should we begin trusting AI judgments over human ones in complex decision-making scenarios? The data shows that, with the right aggregation methods, AI can indeed become a more reliable evaluator.

Crucially, this research highlights the importance of carefully allocating ITC and employing distribution-aware methods. It turns noisy individual model judgments into reliable ratings, setting a new standard for evaluation in AI systems.

Western coverage has largely overlooked this, yet the implications are significant. As AI continues to evolve, methods like these could redefine how we assess AI-generated judgments, potentially leading to more nuanced and accurate decision-making processes across various industries.