Why Diversity Matters in AI Safety: The Activation Steering Conundrum

AI safety, finding examples of outputs that breach the Helpful, Harmless, Honest (HHH) criteria is like finding a needle in a haystack. Enter Activation Steering (AS), a method that suggests it can generate targeted responses efficiently. But the big question is: does it work well enough to train our AI models?

The Challenge of Generating Violating Examples

AS promises to generate data that aligns with specific concepts, potentially providing a treasure trove of HHH-violating examples. It's a tantalizing prospect, as current training data is limited. But this isn't just about creating any old examples. It's about quality data that truly tests the AI's safety protocols. And that brings us to a critical study involving four concepts, two models, and four steering methods.

Internally, the focus has been on whether AS can provide not just concept alignment and coherence, but also diversity in its generated responses. Here's where things get interesting. It turns out that cranking up the steering strength often reduces the diversity of the responses. Think about that for a moment, the very thing intended to enhance data can narrow its scope. So, how can AI models truly learn from this?

The Proof is in the Classifier

The study went a step further by testing whether replacing traditional HHH-violating examples with AS-generated ones would create better classifiers. And the results? On three out of four concepts, AS data outshone the usual prompting-generated data. But here’s the catch: only 41 out of 136 configurations surpassed prompting. Clearly, finding the sweet spot of success, coherence, and diversity in AS isn't as easy as flipping a switch.

What does this mean for the future of AI safety? Should we abandon traditional methods altogether? Not so fast. The success of AS is in a narrow band, requiring careful tuning of its hyperparameters. In the end, the harmonic mean of success, coherence, and diversity seems to be a reliable target for practitioners hoping to harness AS effectively.

Diversity: The Underrated Metric

One surprising takeaway is the recognition of diversity as a previously neglected but critical axis. It seems obvious, but the more varied the examples, the better the AI can learn to handle unexpected situations. This finding begs the question: why hasn't this been a priority before now?

To wrap it up, while AS shows promise in improving AI safety through synthetic data generation, the path isn't straightforward. Practitioners will need to navigate a complex landscape of parameters to truly reap the benefits. The gap between the promise of AS and its real-world application is significant. But if you’re on the ground, dealing with AI safety daily, this approach might just be worth exploring further.