Decoding Mechanistic Mysteries: Architecture, Task and the Curious Flip

Mechanistic studies of sequence models have long treated the architectures of these models as static, predictable entities. Recurrent models, like LSTMs and GRUs, were believed to concentrate their state in readable formats, while attention-based models like Transformers distributed them. What if we've been seeing just half the picture?

Reversing the Roles

A recent study offers a revelation: the same architecture can exhibit drastically different behaviors based on the task at hand. For instance, when comparing Transformer, Mamba, and recurrent models on tasks like Parity and Dyck-k, the state concentration patterns reverse. In Mamba and recurrent models, Parity is concentrated late, while Transformers gradually build it. Curiously, in bounded-depth Dyck-k, the pattern flips entirely.

This isn’t just a quirk. Even after fine-tuning, Mamba-130M and Pythia-160M models demonstrate this flipping behavior, with Pythia experiencing a persistent Dyck bottleneck at 410M parameters. Crucially, this suggests the task itself, not just the architecture, dictates how information is managed.

Task over Architecture

Why does this flip occur? Two competing explanations emerge in the literature: algebraic structure versus computational structure. To untangle them, researchers introduced a third task: non-commutative S3 permutation composition. Surprisingly, S3 aligns with computational structure rather than algebraic commutativity across all five architectures.

Causal interventions provide further clarity. They reveal that linearly readable directions can be functionally necessary, especially at out-of-distribution lengths on tasks like Parity and Dyck. In pretrained models, yet again, the narrative splits. Fine-tuned Pythia Dyck shows a strong bottleneck in middle layers, while Mamba's final layer remains highly readable.

Why Should We Care?

The key finding here disrupts the neat categorization of sequence models by architecture alone. It begs the question, are we focusing too narrowly on architecture? Perhaps we're missing a trick by not considering how tasks dynamically influence these static structures.

The paper's key contribution: probing isn't just about finding where the state is linearly available. It's also about locating the computational bottleneck, which might not align with our assumptions. What’s missing in our understanding is the nuanced interaction between task and architecture. Clearly, mechanistic signatures are a function of both.

As we develop increasingly sophisticated AI models, this insight is invaluable. It challenges us to reconsider how we evaluate model performance. Should benchmarks reflect both task variations and architectural peculiarities? As AI practitioners, it's time to adapt our frameworks and expectations.