STRAP-ViT: A New Defense Against Adversarial Patches in Vision Transformers

Adversarial patches present a unique challenge to Vision Transformers (ViT). These high-contrast regions can disrupt self-attention, causing confident misclassifications. Enter STRAP-ViT, a novel method that addresses this vulnerability without additional training costs.

Adversarial Patches: The Problem

Adversarial patches are designed to mislead models like ViTs by focusing them on deceptive regions. These patches exploit the self-attention mechanism, pulling focus toward themselves and corrupting the class token. The result? Confident but incorrect classifications.

The paper's key contribution is identifying how tokens affected by adversarial noise exhibit different statistical properties than those that aren't. This insight lays the foundation for STRAP-ViT, which uses Jensen-Shannon Divergence to detect these anomalies during the Detection Phase.

STRAP-ViT: The Solution

STRAP-ViT operates in two phases. Initially, it segregates anomalous tokens using a metric based on statistical divergence. Then, during the Mitigation Phase, it applies randomized composite transformations to these tokens, neutralizing the adversarial noise.

Crucially, STRAP-ViT is designed as a non-trainable plug-and-play block. It's compatible with existing ViT architectures for inference, imposing minimal computational overhead. This practical design choice sets it apart from many resource-intensive solutions.

Strong Results Across the Board

Tested on ViT-base-16 and DinoV2 architectures using datasets like ImageNet and CalTech-101, STRAP-ViT shows impressive performance. The solution handles multiple adversarial attacks, including Adversarial Patch, LAVAN, GDPA, and RP2. It achieves solid accuracies within 2-3% of clean baselines, outperforming the current state-of-the-art.

Why does this matter? As models grow more complex, their vulnerabilities also increase. Solutions like STRAP-ViT ensure that advancements in AI security keep pace with model capabilities. It's a step toward more reliable and trustworthy AI systems.

But a question lingers: Can STRAP-ViT, a defense mechanism, keep up with rapidly evolving adversarial strategies? While it's effective now, the arms race between attack and defense in AI continues.