AGORA: Revolutionizing Token Compression for LLM Agents

Token-level extractive compressors have been a staple for language models, but they're floundering large language model (LLM) agents. Despite achieving impressive compression rates, 1.3 to 13.3 times the original size, these methods fail to maintain performance, collapsing to mean rewards of 0.05 or less. The culprit? Action-grammar destruction, where important tokens like identifiers and action verbs are removed, rendering the remaining content useless.

Introducing AGORA

AGORA steps into this breach with a novel approach: step-level compression. It combines a structural prompt parser with an always-keep floor for critical content, alongside a relevance scorer boasting 125 million parameters. This approach is inference-free and delivers compression without imposing extra demands on the LLM. Across various methods, AGORA stands out, retaining at least 75% of uncompressed performance in eight out of nine test cases.

Why It Matters

Here's why you should care. The disparity in performance highlights a fundamental flaw in current compressor strategies. AGORA's ability to retain performance indicates that the problem isn't with compression itself but with how we approach it. By focusing on which parts of the data are essential, thanks to its structural floor and relevance scorer, AGORA achieves adaptive compression that keeps the system functional.

In a world obsessed with reducing data size, AGORA presents a strong case for quality over quantity. Token-level methods might offer impressive compression ratios, but what's the point if they demolish performance? AGORA's approach isn't just about slimming down data but doing so while preserving its utility.

Compression Without Compromise

Critically, AGORA’s success isn't just in its technical prowess. It's in challenging the status quo. Why stick with a method that clearly doesn’t work for LLMs? Change is necessary. AGORA’s architecture shows that with minimal overhead and smart design, performance doesn't have to be sacrificed for compression.

In a field where every percentage point counts, AGORA proves that step-level compression isn't just viable, it's necessary. Clone the repo, run the test, then form an opinion. It's time to rethink how we compress and what truly matters in maintaining functionality alongside efficiency.