The Hidden State Dilemma: Privacy or Utility?

In the pursuit of privacy in AI systems, researchers have hit a stumbling block. Recent tests on 1,536 Gaussian release covariances for hidden-state privacy revealed a stark reality: none provided both moderate utility and privacy when facing an adaptive attacker.

The Privacy-Utility Trade-off

Here's what the benchmarks actually show: attempts at preserving privacy with Gaussian covariances fall flat when utility is also a goal. Notably, a Fisher-ball lower bound proof exposes a vulnerability. Every full-rank Gaussian release at a constant Fisher utility has a direction where the Mahalanobis signal increases linearly with the hidden width. This effectively rules out the notion of uniform Gaussian safety.

Consider the inverse-Fisher release, denoted asΣ*_diag(𝒦). It's a unique minimax-optimal diagonal mechanism at the first-order KL budget, boasting a worst-attacker top-1 error of ≤ 0.001 across a 32-layer model grid. But, strip away the marketing and you get a solution perched precariously on the edge of privacy and utility, without filling the gap in between.

Mechanisms Under Pressure

The reality is, promising mechanisms like the generalized-eigen mechanism tout a 13x Pareto improvement under Euclidean retrieval. Yet, they collapse under pressure, with a 100% top-1 error facing an adaptive Mahalanobis attacker. It's a bleak scenario when a full-trajectory sequence inverter can recover 94% of clean GPT-2 prefixes, but fails completely underΣ_diag.

So, where do we stand? A split-memory transformer trained from scratch manages a Mahalanobis range of 20 to 33 at 90 million parameters, maintaining a 6 to 24x advantage over similarly budgeted GPT models. Pretrained counterparts peak at a meager 9.3. This performance dichotomy raises an important question: is it time to rethink the architecture rather than fixate on release mechanisms?

Rethinking Hidden-State Release

The numbers tell a different story, pushing the conversation from mechanism design within the Gaussian framework to a broader architecture or release co-design approach. Should the focus shift to innovative architectures that inherently balance privacy and utility? It seems increasingly likely as traditional methods fail under scrutiny.

The stakes are high. As AI continues to integrate deeper into sensitive domains, the race for a solution that harmonizes privacy and utility in hidden-state releases isn't just academic. It's a pressing challenge that needs addressing now.