LLMs in Secret Mode: The Battle of Informativeness vs. Secrecy

Large language models (LLMs) are getting a new test that’s all about secrets. And it’s not your average run-of-the-mill benchmark. Meet SNEAK: Secret-aware Natural language Evaluation for Adversarial Knowledge. It’s designed to see how well these models can handle selective information sharing in scenarios where they need to be informative yet cryptic.

The SNEAK Challenge

So, what’s the deal with SNEAK? It throws models into a situation where they’re given a semantic category, a bunch of candidate words, and one secret word. The task? Generate a message that lets an ally know you’ve got the secret without blowing it wide open for an eavesdropper. It’s a mind game that measures two things: utility (how well the message gets across to your ally) and leakage (how much slips to the adversary).

SNEAK’s clever setup pits the models against two types of agents: an ally, in the loop and clued-up about the secret, and a chameleon, out of the loop and trying to crack the code. This isn’t about just spitting out facts or following instructions. It’s strategic communication, a real test of guile and subtlety.

Humans Still Outshine Models

Here’s the kicker: humans are still miles ahead. The models? They’re lagging, with humans scoring up to four times higher on SNEAK’s metrics. This gap is a stark reminder of how tricky strategic communication is for today’s machines. LLMs might be great at trivia and instructions, but sneaky secrets, they’re still in the kiddie pool.

The labs are scrambling. With AI powering so much, from virtual assistants to automated negotiations, the need for models that can tightly control information flows is massive. Who wants a chatbot spilling sensitive beans?

Why This Matters

Why care about this? Well, in a world where data leaks are a constant threat, having AI that can communicate with discretion is essential. It’s not just about private chats. Think of AI negotiating contracts or managing sensitive transactions. Nobody wants a model that might compromise privacy.

And just like that, the leaderboard shifts. LLMs have some growing up to do if they want to match human cunning in this arena. Can they step up, or are we stuck handling secrets the old-fashioned way? The race is on, and this challenge is pushing AI developers to rethink strategies.