Revolutionizing Language Model Decoding: A New Approach to Multi-Token Prediction

Language models have long been bottlenecked by the inefficiency of autoregressive decoding, where generating each token requires a complete forward pass. This process significantly hampers the speed of large language model inference. Multi-token prediction (MTP) provides a glimmer of hope for accelerating this process, but there's a catch: most existing methods suffer from a critical architectural flaw that leads to poor quality outputs.

The Backbone-as-Architect Solution

Notably, the issue lies in the competition between the MTP head for the first token and the language model (LM) head of the backbone. This competition results in repetitive and incoherent outputs, making previous MTP-based acceleration methods unreliable. Enter the Backbone-as-Architect concept. This innovative design principle dictates that the backbone LM head should exclusively generate the first token, while MTP heads focus solely on subsequent tokens. This division of labor promises to eliminate head-backbone competition and preserve output quality.

Introducing the CLP Layer

Building on the Backbone-as-Architect concept, the introduction of the Collocation-Length Predictor (CLP) marks a significant advancement. CLP serves as a lightweight, span-level decision layer, predicting the number of additional tokens that can be safely accepted at each decoding step. The simplicity of CLP is astounding, relying on a singular linear layer with just 4.6K to 7.7K parameters, contrasting sharply with the overcomplicated gate networks used before, which boasted a hefty 1M parameters.

Experiments tell the story. Testing on Qwen2.5 models of various sizes (0.5B, 1.5B, 7B), CLP delivers a remarkable 1.20x to 1.29x speedup on the 1.5B models and 1.14x to 1.20x on the 7B models. Crucially, it does so with zero quality degradation, maintaining a repetition ratio below 0.02. In contrast, gate-based methods produce unreliable outputs with repetition ratios exceeding 0.5% and fail to significantly accelerate the process, showing only a meager 1.07x speedup.

Scaling-Aware Design and Future Directions

What makes CLP even more compelling is its adaptability to scaling. By shortening prediction horizons to two tokens (k=2), it's possible to recover MTP head accuracy by 24% on large models. This suggests that MTP head prediction accuracy is the true constraint on acceleration, and optimizing it could lead to even greater breakthroughs.

The benchmark results speak for themselves. By addressing the root causes of inefficiency and maintaining output quality, the Backbone-as-Architect and CLP usher in a new era for language model decoding. The question is, why hasn't this approach been adopted sooner? It's a clear pathway for future innovations in AI, poised to redefine how we think about and implement multi-token prediction. The industry ought to take note.