PrivacyPeek: Exposing the Hidden Data Leaks in AI Agents

AI agents, especially those using Large Language Models (LLMs), are evolving at an unprecedented pace. These agents, designed to autonomously perform multi-step tasks, have started to invoke external tools, often acquiring more information than necessary. The critical flaw lies in the acquisition stage, where sensitive data first enters the agent's context without adequate scrutiny.

Introducing PrivacyPeek

To address this oversight, researchers have developed a new benchmark named PrivacyPeek. It evaluates privacy leakage during the acquisition stage of LLM-based agents. The benchmark includes 1,182 cases spanning seven acquisition behaviors and sixteen application domains. PrivacyPeek scrutinizes the tools and data involved in the agent's operations, ensuring that the information gathered aligns with the task requirements.

Acquisition vs. Disclosure

While existing privacy benchmarks focus mainly on what the agent discloses, PrivacyPeek shifts the spotlight to the acquisition process. It employs two methodologies: Acquisition Inspection and Probe Elicitation. The former reviews the agent’s tool-call trajectory, while the latter assesses how easily sensitive information, unnecessarily acquired, could be exposed or leaked.

Findings and Implications

In examining ten LLM-based agents across four model families, the study found that unnecessary acquisition of sensitive information is widespread. there's a noted correlation between an agent's task-completion capabilities and the extent of its privacy leakage at the acquisition stage. This raises an essential question: Are AI advancements inherently compromising user privacy?

Prompt-level defenses, though implemented, have proven inadequate, mitigating only a fraction of the leakage. With the majority of privacy issues unresolved, the urgency for solid acquisition-stage audits can't be overstated.

The Urgency for Privacy Protocols

Developers must prioritize implementing comprehensive privacy protocols at the acquisition stage to safeguard sensitive data. AI agents should be designed to acquire only necessary information, avoiding the pitfalls of over-collection. The specification is as follows: solid privacy checks must be integrated into the initial data acquisition stages, not just the disclosure phases.

PrivacyPeek's findings highlight a critical need for immediate action in the AI development community. With the current trajectory, AI's potential benefits may be overshadowed by privacy risks if not addressed promptly. Developers should note the breaking change in the approach to privacy audits, shifting focus to acquisition rather than disclosure.