The Geometric Dance of Transformers: A Study in Divergence and Commitment

Decoder-only transformers, when pushed to the limits of their capacity, exhibit a fascinating dance. They process factual queries by exploring divergent paths through hidden-state space. These pathways, when faced with correct and incorrect single-token continuations, reveal a unique geometric signature.

The Divergence Revealed

When tasked with a factual query, a transformer keeps its displacement vectors at a steady magnitude. But it's the direction that's intriguing. These vectors rotate apart, showcasing an angular separation that intensifies in mid-depth, only to resolve asymmetrically in the later layers. This outcome is startling: the model ends up favoring the incorrect token by a factor of 11.5 times more than the correct one. It's a quirk that’s consistent across six decoder-only transformers, spanning architectures with parameters from 1 billion to 13 billion.

Why does this matter? Because it suggests that what looks like a model rejecting a wrong continuation is actually a complex geometric pattern. Yet, this isn't a causal account but rather an observational one. It raises the question: is AI merely a puppet of its mathematical underpinnings, or do these patterns hint at something greater, perhaps even an emergent form of reasoning?

Beyond Simple Explanations

Interestingly, one model, the Qwen2 at 1.5 billion parameters, bucks the trend. Its flat profile under current extraction protocols might be due to a tokenizer-fragmentation artifact rather than a real limitation. This raises more questions than it answers. Is there a scale threshold where these patterns emerge, or is Qwen2 an outlier?

Single-layer activation patching fails to recover the correct token, implying that the late-layer asymmetry isn't tied to a single component. This challenges the straightforward localized-recall narrative. It supports the notion of a distributed-by-trajectory processing structure. The inference isn't confined to one layer, it’s a cumulative affair across many.

The Bigger Picture

If these observations hold, they could redefine our understanding of AI reasoning. The traditional view of AI as a machine simply regurgitating learned patterns might need an upgrade. Perhaps AI decision-making is more about navigating a geometric landscape than we previously assumed. The intersection is real. Ninety percent of the projects aren't, but this one might be onto something significant.

The question remains: can we trust AI not to spiral into error when it inherently favors incorrect continuations so heavily? If the AI can hold a wallet, who writes the risk model?