PERSUASIONTRACE: Rethinking How Machines Influence Human Beliefs

Large language models (LLMs) are reshaping how we think, especially in high-stakes scenarios. Yet, traditional studies often focus only on whether persuasion occurred, not on the intricate shifts within a conversation. Enter PERSUASIONTRACE, a framework shedding light on the finer details of persuasion in human-LLM interactions.

A New Framework for Understanding Persuasion

PERSUASIONTRACE isn’t just a tool. It's a comprehensive system designed to study persuasion across multiple dialogue turns. Unlike traditional pre/post belief assessments, this framework records the subtle movements of beliefs throughout an interaction. It annotates each persuader's turn with rhetorical elements like logos, pathos, and ethos, providing a granular look at persuasion dynamics.

Why does this matter? Because it changes how we evaluate persuasive systems. Rather than just ticking boxes on whether a belief shift occurred, PERSUASIONTRACE offers a process-level evaluation. This means a deeper, more scientific understanding of how persuasion unfolds. In a world where AI has the power to shape opinions, that's not just important, it's essential.

Human-like Persuasion Dynamics

Using PERSUASIONTRACE, researchers discovered that humans cluster into two distinct groups multi-turn belief updates. They also found that people are susceptible to various rhetorical strategies. What’s more, LLMs proved persuasive across a variety of topics and formats, from text to audio, and in multi-turn dialogues.

But here’s the kicker: standard LLMs, which have been used to simulate human persuasion targets, fail to capture the complexity of human belief dynamics. That’s where PERSUASIONTRACE’s innovative Bayesian-network simulated target comes in, maintaining a latent belief state over time to reflect more cognitively realistic updates.

Implications for Future AI Systems

The findings show that the Bayesian target scored an impressive 81 compared to a human reference score of 80. In contrast, baseline LLM targets lagged behind with a score of 64. This isn’t just a technical detail, it's a significant milestone. The AI-AI Venn diagram is getting thicker, and with it, the need for safer and more effective persuasive systems.

So, what does this mean for the future of AI-human interactions? If we’re building machines that can persuade, it’s essential to understand not just the outcome, but the process. PERSUASIONTRACE provides a stronger base for optimizing these systems safely. But if agents have wallets, who holds the keys to this persuasive power?

The convergence of AI capabilities with human cognitive processes demands an ethical framework. As machines take on more agentic roles, understanding their influence, and controlling it, becomes not just a technical challenge but a societal imperative.