Tackling the Coverage Illusion in RAG Pipelines

In the field of retrieval-augmented generation (RAG) pipelines, the industry has been caught in a costly dilemma. The go-to methods for query augmentation, like HyDE and query expansion, are applied universally. Yet, this approach leads to hefty large language model (LLM) inference costs and increased latency. The burning question: Is this cost justified?

The Case Study

A recent case study by the Danish National Encyclopedia sheds light on the so-called 'Coverage Illusion.' Evaluating 20,000 query-workflow pairs, it turns out that synthetic queries suggest LLM augmentation is needed for over 90% to achieve optimal retrieval coverage. Yet, in production, only 27.8% of real user queries require such augmentation. This gap points to a structural mismatch between synthetic and real query distributions. A striking revelation, isn’t it?

Beyond Pre-retrieval Routing

The study reveals that pre-retrieval routing alone can't bridge this gap. Why? Because the need for LLM augmentation only becomes apparent after searching the index. Testing across four machine learning paradigms confirmed this. The solution isn't in pre-retrieval. It's in a post-retrieval cascade system.

This system proposes running workflows in a cheapest-first order, escalating to LLM augmentation only when necessary. With no added training overhead or secondary infrastructure, this cascade improved quality by +0.140 Composite Overall points over the Always-HyDE model and slashed latency by 31.8%. Impressively, it handled 72.2% of real queries sans LLM augmentation.

Why It Matters

Here's the crux: many systems could be squandering resources by augmenting every query. The practical implication is clear. Systems need to adopt smarter approaches, like the post-retrieval cascade, to optimize performance without unnecessary costs.

Could it be that your system is over-engineered? Perhaps it's time to reconsider. Query augmentation should be judicious, not blanket. The data from the Danish case study is a wake-up call for developers: your LLM resources might be better spent elsewhere. Read the source. The docs are lying.