Rethinking Decoding: How Anchor-based History-stable Decoding Outshines Its Peers

In the ever-competitive landscape of large language models, Diffusion Large Language Models (dLLMs) have recently emerged as a viable alternative to the long-established autoregressive models. While Semi-autoregressive (Semi-AR) decoding has been the go-to strategy for its perceived superior performance, it turns out there's a catch. Its block constraints have been causing unnecessary delays in decoding, particularly for cross-block stable tokens. Now, isn't it time we question the effectiveness of this much-touted methodology?

The Inherent Flaws in Semi-AR Decoding

Let's apply some rigor here. Our analysis of Semi-AR decoding reveals that its inherent block constraints are more of a hindrance than a help. The decoding of stable tokens across blocks is delayed, which is far from ideal in the quest for efficient model performance. The industry has been clinging to this method despite these shortcomings. The claim doesn't survive scrutiny when you consider that naive lookahead decoding often turns out to be unreliable. Token stability, we find, is more closely tied to convergence trends while historical information seems to be isolated rather than integrated.

A New Approach: Anchor-based History-stable Decoding

Enter Anchor-based History-stable Decoding (AHD), a fresh strategy that aims to turn the tables. AHD is a training-free, plug-and-play dynamic decoding approach that monitors the stability trend of tokens through dynamic anchors. Once a token stabilizes, it triggers early cross-block decoding, boosting both performance and efficiency. It's a simple yet profound change. This method doesn't just tweak the process. it redefines what's possible by cutting through the noise and directly addressing the bottleneck in Semi-AR decoding.

Performance Gains That Matter

Color me skeptical, but the numbers are compelling. Our extensive experiments in language, vision-language, and audio-language domains show that AHD improves both performance and inference efficiency. For example, on the BBH benchmark, AHD reduces decoding steps by a staggering 80% while improving performance by 3.67%. Such results aren't mere anomalies. they're a testament to a methodology that embraces change and seeks optimization where it counts.

So, what does this all mean for the future of language models? The writing's on the wall: methodologies that resist adaptation will eventually fall by the wayside. The industry should take note and consider AHD not just as a tweak, but as a viable path forward in the evolution of decoding strategies.