Decoding the Identity Trap in EEG Models

EEG foundation models, accuracy isn't always what it seems. The challenge lies in distinguishing genuine clinical biomarkers from subject-specific features. Enter the Identity Trap. This dilemma questions whether high accuracy under subject-disjoint cross-validation truly reflects clinical insights or just subject-identity correlations.

Introducing FMScope

To address this, researchers propose FMScope, a diagnostic protocol aimed at identifying the Identity Trap at the representation level, before any fine-tuning occurs. FMScope uses five diagnostic tools: variance decomposition, subject-axis erasure, aperiodic 1/f ablation, layer-wise label probing, and within-subject direction consistency. These tools were tested on three pretrained EEG models, LaBraM, CBraMod, and REVE, across four datasets.

Key Findings

The results were telling. The Identity Trap is pervasive, with frozen subject-variance exceeding a random null between 13 to 89 times in all model-data pairs. Fine-tuning exacerbates this, increasing variance by 10 to 63 percentage points. However, this subject-identity dominance can be mitigated. By erasing the linear axis associated with subject identity, label decoding accuracy improved by 6 to 12 points in primary cells, and by 4 to 27 points across external cohorts.

Aperiodic 1/f signals were identified as a carrier of subject identity, resulting in a 9 to 19 point drop in subject probe scores when removed from LaBraM and CBraMod. Interestingly, REVE showed complete saturation of subject identity without reliance on aperiodic signals. This raises the question: how much of what we measure is genuinely clinical versus just noise?

Why It Matters

The Identity Trap highlights a critical flaw in shortcut learning within EEG models. When models latch onto subject-specific cues rather than true clinical markers, it skews outcomes and misleads researchers. The fact that subject-disjoint splitting fails to eliminate this bias underscores the complexity of the issue.

FMScope stands out by isolating true clinical markers from subject identity. The ability to do so is vital. If we can't trust our models to pinpoint real biomarkers, the entire premise of clinical EEG analysis is on shaky ground. Slapping a model on a GPU rental isn't a convergence thesis. It's important we separate signal from subject identity if these models are to contribute meaningfully to clinical practice.

Ultimately, the promise of EEG foundation models hinges on overcoming the Identity Trap. Until then, the question remains: Are we diagnosing patients or just the quirks of our models?