Unlearning Machines: The Challenge of Causal Forgetting

Machine unlearning is a growing field tackling the removal of knowledge from AI models. But here's the rub: current evaluations are skewed, especially Why-type questions. These questions, which probe causal and relational knowledge, make up a minuscule fraction of major benchmarks. Specifically, they're less than 0.06% of CounterFact and a mere 0.6% of ZSRE. This lopsided representation allows methods to perform well overall, while failing at causal unlearning.

The Benchmark Problem

To address this, the 5WBENCH was introduced, offering a balanced approach with 5,000 samples covering Who, What, When, Where, and Why. With 1,000 examples for each category, it's the first benchmark to quantify failures in causal unlearning. Strip away the marketing and you get to the numbers: no current method achieves both high forgetting and retention for Why-type questions. Why? Because these questions often require complex multi-hop reasoning and spread across long answer spans.

MAAT: A New Approach

Enter MAAT (Multi-phase Adapter-Aware Targeted Unlearning), a novel framework designed to tackle this exact issue. It uses a three-phase process involving gradient-projected ascent and a mix of techniques like SVD rank-dimension pruning. The result? MAAT is the first to achieve high levels of both forgetting and retention on causal knowledge. It sets a new operating point on the forget-retain Pareto frontier.

Why Should You Care?

So why does this matter? The reality is, as AI becomes more integrated into decision-making processes, understanding how effectively it forgets information is important. If a model can't forget causal knowledge accurately, it risks retaining biases. Frankly, the architecture matters more than the parameter count unlearning. As AI systems continue to evolve, benchmarks like 5WBENCH and methods like MAAT will be indispensable tools for ensuring these systems are both accurate and fair. Who would want an AI that remembers the wrong things?