Decoding Transformers: A Journey Through Layers

Understanding the intricacies of transformer models has always been a challenge. It's not just about what these models encode but how their internal representations change across layers. Recent insights offer a fresh perspective, positioning the forward pass of transformers as a journey, a trajectory through a high-dimensional space.

The Geometry of Thought

This isn't about seeking pre-defined features. Instead, the focus is on the geometry of these trajectories, measured using five distinct metrics: trajectory length, curvature, a semantic convergence index, layerwise cosine similarity, and representational stability. These metrics, applied across models like GPT-2, TinyLlama, and Qwen2.5, uncover patterns hidden within the complex layers of these neural networks.

What do these metrics reveal? For starters, semantically related prompts show significant convergence in middle-to-late layers, pointing to attractor-like dynamics with a peak convergence index between 0.41 and 0.58. These aren't trivial numbers, they suggest a nuanced understanding of semantic processing that's not just emergent but structured.

Curvature and Complexity

A standout finding is the role of trajectory curvature. Reasoning tasks exhibit greater curvature than simple lexical tasks, with figures ranging from 0.71 to 0.83 radians compared to 0.27 to 0.31 radians. It's a revelation: curvature might encode computational complexity, offering a tangible measure of how transformers handle different types of information.

Consider ambiguous tokens. These elements cause trajectory bifurcation, leading to a 5.6x increase in representational separation by the final layer. This separation is absent in clear-cut controls, underscoring how ambiguity is processed at a structural level.

Universal Patterns and Open-Source Tools

The journey doesn't stop there. Layerwise cosine similarity reveals a universal three-phase process across models: encoding, elaboration, and output preparation. This tri-phase structure holds true regardless of the architecture, whether it's GPT-2 or Qwen2.5. Interestingly, these effects disappear when layers are shuffled or embeddings randomized, emphasizing the inherent order within these networks.

With the release of a fully open-source, model-agnostic pipeline, the AI-AI Venn diagram is getting thicker. Trajectory geometry provides a probe-free, principled approach to mechanistic interpretability. But here's the big question: Are we finally on the brink of truly understanding AI's decision-making processes, or are we just scratching the surface?