Boosting LLM Efficiency with Reset-and-Discard Method

Large language models (LLMs) have long been evaluated on their ability to answer questions correctly using a metric known as pass@k. However, this measure doesn't always reflect real-world constraints where budgets and resources are limited. Instead, a metric like coverage@cost, which considers the average number of unique questions answered relative to total attempts, could provide a more accurate picture.

Introducing the Reset-and-Discard Method

The paper, published in Japanese, reveals a new method called Reset-and-Discard (ReD), designed to tackle inefficiencies in LLM performance. This method offers a way to increase coverage@cost regardless of the pass@k form. Essentially, ReD aims to maximize efficiency, enabling LLMs to answer more unique questions with fewer attempts.

Notably, the benchmark results speak for themselves. Experiments conducted on three different LLMs, focusing on coding using HumanEval, math with GSM8K, and reasoning through MMLU-Pro benchmarks, demonstrate that ReD can significantly reduce the number of attempts needed to achieve desired coverage. This translates to reduced computational resources and lower costs, both tokens and USD.

A New Perspective on Measuring LLMs

What's particularly intriguing is ReD's ability to infer the power-law exponent of pass@k if it's unavailable. This implies a deeper understanding of how LLMs process information and predict outcomes. But here's the key question: why hasn't coverage@cost been the standard all along?

Western coverage has largely overlooked this nuanced approach, focusing instead on the more straightforward pass@k. But compare these numbers side by side, and you'll see a clear advantage in adopting methods like ReD for LLM assessments.

Why ReD Matters

The adoption of ReD could fundamentally change our approach to evaluating LLMs. By reducing the resources needed to achieve efficient coverage, companies can lower operational costs and improve model performance. In an era where AI is becoming increasingly central to business operations, these savings could be significant.

ReD maintains its effectiveness even in scenarios with imperfect verifiers, outperforming tested allocation baselines. This robustness suggests that ReD could become a standard method in optimizing LLMs across various applications.

The data shows that the Reset-and-Discard method represents a strategic shift in AI performance evaluation. As AI models continue to grow in parameter count and complexity, methods like ReD will be important in ensuring these models are both cost-effective and high-performing.