Reimagining Two-Bit Quantization: A Fresh Take on LLM Inference

Memory-efficient inference in large language models (LLMs) is a hot topic, and two-bit weight quantization is at the center of this discussion. Traditionally, the standard W2 level set {-2,-1,0,+1} has been the go-to. But in certain settings like W2A4/KV4, this approach often collapses, leaving room for innovation.

A New Level Set

Enter the concept of asymmetric W2. It offers substantial improvements over the traditional set, suggesting that the issues with W2A4 aren't just about bit-width. Instead, there's a deeper reconstruction-level problem at play. In models like LLaMA-2-7B and LLaMA-3.1-8B, the pretrained weights are nearly zero-centered. By employing Hadamard rotation, these weights transform into a Gaussian-like shape, reducing excess kurtosis and Q-Q error dramatically.

The Birth of Qift

Based on this transformation, a new proposal called Qift emerges. Qift utilizes a fixed no-zero W2 level set for rotated W2A4/KV4 inference. Its main level set is {+/-0.5, +/-1.5}, which can be reparameterized to {+/-1, +/-3}. Notably, this approach doesn't require training, learned codebooks, or group grids. It even retains the standard per-channel scale. By focusing on the effective inner/outer centroid ratio range of 0.25 to 0.33, Qift demonstrates why certain methods like mirror no-zero (MNZ) work better than the traditional {+/-1, +/-2} set.

Why It Matters

In practice, the no-zero level sets consistently outperform the standard W2 level set in various metrics, from pure W2A4 perplexity to downstream accuracy. At L=16 mixed precision, they close the gap to W3A4 significantly, while keeping half of the transformer layers at two-bit precision. The farmer I spoke with put it simply: It's a straightforward, deployment-friendly alternative to more complex learned W2 codebooks.

So, why should you care? Because this isn't just about tweaking a few numbers. It's about redefining how we approach efficiency in model inference. The story looks different from Nairobi, where every bit of memory counts and efficiency could mean the difference between deployment and stalling in the development phase. Automation doesn't mean the same thing everywhere, and AI, these changes could be vital for emerging economies where resources are limited.