Breaking Down Equivariance: Striking the Balance in Neural Networks

Equivariance has long been heralded as the holy grail for neural networks, enhancing generalisation and ensuring physical consistency. Yet, in a surprising turn, non-equivariant models are making a comeback, driven by their superior runtime performance. This resurgence challenges the notion that perfect symmetry is always the best approach in real-world applications where imperfections abound.

Rethinking Equivariance

The dilemma lies in balancing symmetry with efficiency. Recently, approximately equivariant models have emerged as contenders for this middle ground. These models aim to reconcile respect for symmetries with a flexible fit to the data distribution.

Traditionally, achieving approximate equivariance involved sample-based regularisers, often demanding extensive data augmentation during training. This approach notably struggles with continuous groups like SO(3) due to high sample complexity. The result? A bottleneck in model efficiency and performance.

Projection-Based Regularisers to the Rescue

Enter projection-based regularisers. Unlike their predecessors, these methods use the orthogonal decomposition of linear layers to separate equivariant components from non-equivariant ones. This shift penalises non-equivariance at an operator level instead of focusing on individual points, thereby covering the full group orbit.

The mathematical elegance of this framework lies in its ability to compute the non-equivariance penalty both exactly and efficiently, spanning the spatial and spectral domains. It's a move that promises not just theoretical robustness, but practical runtime gains.

Why It Matters

In the race to improve AI models, performance can't be sacrificed at the altar of symmetry. As new methods like projection-based regularisers outperform prior approaches, they offer a compelling alternative. They deliver both enhanced model performance and efficiency, without the exorbitant costs associated with data-heavy regularisers.

But here's the real question: in a world obsessed with symmetry, are we finally ready to accept that imperfection might just be the key to unlocking better neural networks? If the AI can hold a wallet, who writes the risk model? It's a provocative thought, and one that the industry can't ignore.