Rethinking Retrieval: The Case for Portfolio-Based Systems

Retrieval-augmented generation (RAG) systems have long relied on a single retriever, operating under the assumption that one set of hyperparameters can tackle a wide range of queries. But let's face it, not all questions are created equal. From straightforward factoids to intricate multi-hop reasoning, the diversity of queries demands a more nuanced approach. Enter the portfolio of retrievers.

Why a Portfolio?

The concept of a retriever portfolio isn't just about diversifying for the sake of it. It's about strategically selecting a small, yet diverse, set of retrievers from a vast pool to effectively cover different regions of the target query distribution. This isn't slapping a model on a GPU rental. it's a calculated convergence strategy. A portfolio that anticipates the best-of-k scenario across the query distribution can offer near-optimal performance.

In practical terms, this approach is showing promise. Across various QA benchmarks, these learned portfolios coupled with a router pipeline consistently outrace the traditional single-retriever systems. The results aren't just better in retrieval metrics but also in the quality of answers. It's evidence that a one-size-fits-all model might not be the best fit after all.

Beyond the Traditional

But there's more to this than just outperforming existing systems. Fixed portfolios offer distinct advantages, particularly efficiency. They enable parallel retrieval and LLM calls, which reduces latency and token cost without compromising accuracy. In some cases, the results are even better. So, if the AI can hold a wallet, who writes the risk model?

This new method also challenges the notion that hyperparameter tuning at inference-time is the pinnacle of efficiency. The traditional approach might yield good results, but it's akin to playing catch-up during the game. A well-constructed retriever portfolio is like having your players already on the field, ready to score.

The Future of RAG Systems

The implications for RAG systems are clear. Instead of doubling down on a single retriever, it might be time to embrace a more dynamic, multi-retriever framework. This isn't just another AI-AI project. It's a tangible solution to a real problem. The intersection is real. Ninety percent of the projects aren't.

So, where do we go from here? The idea of a retriever portfolio isn't just a theoretical construct. It's a practical advancement that could reshape how we handle the growing complexity of AI queries. It's not about whether this change will happen, but how soon.