The Surprising Geometry of Neural Networks and Human Perception

Adversarial attacks remain a thorn in the side of machine learning experts, but why do these seemingly innocuous perturbations wreak such havoc on neural networks? The answer might lie in the surprising geometric properties of these networks.

The Perceptual Manifold Conundrum

Neural networks assign inputs to categories based on what's known as the perceptual manifold (PM). This manifold represents the space of all inputs the network confidently identifies as belonging to a particular category. Intriguingly, the dimensionality of these PMs in neural networks dwarfs that of human concepts.

What's the big deal with dimensions? As dimensions increase, the volume of space grows exponentially. This exponential growth signals a significant mismatch between how machines and humans perceive concepts. If machines label exponentially more inputs under certain concepts compared to humans, the implications for machine learning are serious.

Adversarial Examples: A Natural Byproduct

Adversarial examples, those small but potent perturbations that mislead networks, might be a direct consequence of this dimensional disparity. When a network's PM covers vast swathes of the input space, virtually any input is near a class concept's PM. This proximity makes it easy for adversarial examples to emerge, effortlessly fooling the network.

Here's the kicker: achieving adversarial robustness likely demands aligning the dimensions of machine and human PMs. strong accuracy, the metric for evaluating a model's resilience against such attacks, appears negatively correlated with PM dimensionality. But can we realistically bring these dimensions in line?

The Path to Alignment

Across 18 different networks, even the most strong models showed significant misalignment. Only those whose PM dimensionality approached that of human concepts showed any real alignment with human perception. This finding suggests that the 'curse of dimensionality' in machine PMs is a formidable barrier to achieving true robustness against adversarial attacks.

The AI-AI Venn diagram is getting thicker. The convergence of alignment and adversarial research indicates a promising yet challenging avenue. How do we bridge the gap between machine inference and human perception? If we can solve this, the implications for AI's reliability and safety are monumental.

In the end, we're building the financial plumbing for machines, but perhaps we should first tackle the perceptual plumbing. The collision between high-dimensional inference and human-centric concepts isn't just a technical challenge, it's a frontier that might redefine how we trust and deploy AI systems.