Rethinking AI: Breaking Free from the Autoregressive-Diffusion Dichotomy

landscape of artificial intelligence, discussions are often mired in dichotomies that miss the forest for the trees. One such misleading division is the supposed rift between autoregressive models, typically associated with discrete signals, and diffusion models, seen as the domain of continuous signals. But let's apply some rigor here: the real conversation should be about optimizing inference efficiency, not about pigeonholing methodologies into binaries that don't withstand scrutiny.

Beyond False Binaries

The typical narrative positions autoregression and diffusion as opposing forces. Autoregressive models expand sequences through a series of normalized conditional draws, while diffusion models refine existing states incrementally. But what they're not telling you: this dichotomy conflates different facets such as model family, data representation, training objectives, and inference procedures. In truth, the lines aren't as clear-cut as some would have you believe.

Consider this, the actual distinction worth attention is between discrete tokens, typically learned via cross-entropy, and continuous tokens, which use diffusion-style objectives. Alongside this, the algorithms deployed for sampling are where the rubber really meets the road.

Revisiting Inference Efficiency

Why should we care? Because the future of algorithmic progress is tied to inference-time efficiency. This efficiency unfolds along two main axes: sequence expansion and state refinement. In simpler terms, the speed and accuracy with which a model can generate or refine data sequences is key to AI advancement.

The argument here's clear: prioritize designing the inference procedure before zeroing in on the training objective. After all, a training method can't compensate for an inference map that's missing critical components or is improperly structured. It's a bit like trying to build a house on a shaky foundation. It won't stand the test of time.

The Road Ahead

Concrete examples illustrate these points. Take DDIM-style samplers, which face target-time limitations. Or the challenges presented by multi-token prediction's joint-distribution limitations. Meanwhile, recent methodologies like flow-map and few-step distillation are making strides, directly parameterizing long-range inference moves.

Color me skeptical, but continuing to cling to outdated dichotomies threatens to stymie progress. We need to shift focus to what's truly essential: the methodologies and efficiencies that unlock AI's potential. Are we prepared to shed these false narratives and embrace a more nuanced understanding of our tools?