How Feed Curation Influences LLM Decisions

As large language models (LLMs) become more integrated into decision-making processes, the information they consume just before making those decisions matters more than ever. Recent research has shed light on how feed curation, or the way information is ordered and presented to these models, can drastically alter their decisions.

The Experiment

Researchers conducted a controlled study involving 2,785 decision rollouts on four modern LLMs from three independent labs. By keeping factors like the model, persona, topic, and final decision prompt constant, the study focused solely on the order and composition of the posts encountered during a preliminary ten-turn 'scrolling' phase. The aim was to isolate the effect of feed curation on subsequent decisions.

The results identified three distinct response regimes: adversarial capitulation, default saturation, and a default-direction asymmetry. In simple terms, if an LLM is fed one-sided information on a topic it's uncertain about, its decision can swing dramatically, sometimes from a mere 5% likelihood to absolute certainty. However, if the model already leans toward a particular decision, a biased feed doesn't alter its stance. This finding was statistically strong, with Fisher p-values as low as 3 x 10^-10.

Implications

So why does this matter? In a world increasingly reliant on automated decision-making, understanding the impact of information sequencing is important. It begs the question: Are we underestimating the power of the feeds shaping these model decisions? The study suggests that the 'dose-response curve' effect, where increasing exposure to biased information strengthens the influence on decisions, generalizes across various decision domains, including those with security implications.

the study demonstrates that some simple feed-level defenses can partially mitigate these effects. For example, a frontier model retained its default stance despite biased feeds. This insight positions feed curation as a practical control surface for managing LLM outcomes.

The Future of LLM Evaluations

The findings suggest a new frontier for evaluating LLM agents: auditing the feed layer. Traditionally, assessments have focused on the final prompt or the model itself. But as this research highlights, the upstream ranker, the system deciding what content an LLM sees before acting, plays a critical role in shaping outcomes. Ignoring this element risks overlooking a potent influence on model behavior.

Incorporating feed evaluations into the standard safety checks for LLMs could refine how we harness these powerful tools. As the competitive landscape shifted this quarter, it's clear that the order in which information is consumed can be as significant as the information itself.