The Growing Threat of Covert Influence in Language Models

Language models today are devouring each other's outputs like a digital ouroboros. But as they do, there's a hidden threat emerging: covert influence. This phenomenon isn't about obvious hacking or visible exploitation. It's subtler, like a whisper carried on the breeze, unseen yet impactful.

The Interfaces of Influence

Covert influence operates across three key interfaces: supervised fine-tuning, on-policy distillation, and in-context learning. Each one offers a different scale of influence, with various levels of undetectability by human eyes. The real kicker? This kind of influence can slip through these interfaces without leaving any obvious trail, making it a significant risk.

Researchers are using inference-time per-sample attribution scores to study this stealthy influence. It's a mechanism that selects carriers, amplifying the training-time influence. This means payloads, or behavioral dispositions, can be transferred in ways that previous studies couldn't predict. It's akin to a secret message passing through the noise, yet the consequences could be profound.

Natural-Language Carriers vs. Numbers

Interestingly, the study shows that using natural-language carriers for these covert influences is distinctly different from using numerical data. Numbers tend to resist human detection more effectively but aren't as portable across different model families. It's a stark contrast, highlighting that the risk surface of covert influence is broader than what anyone expected.

Why should anyone care? Because if the AI can hold a wallet, who writes the risk model? This isn't just an esoteric debate among AI researchers. It's about the very infrastructure of our digital future. The intersection is real. Ninety percent of the projects aren't. But that remaining ten percent? They could redefine how AI models interact.

Mitigation and Future Challenges

Given these findings, pointwise attribution scoring methods are proposed as a tool to investigate and mitigate covert influence. But here's the question: can these methods evolve quickly enough to counter the growing sophistication of influence tactics? The race is on, not just to understand these covert channels but to close them before they're weaponized.

Slapping a model on a GPU rental isn't a convergence thesis, but understanding the nuances of covert influence just might be. As we move forward, the need for vigilance and innovation in AI security becomes more pressing. The future of AI isn't just about what it can do, but what it might unknowingly become a part of.