Latent Terms: Unlocking Sparse Retrieval from Dense Models

Dense retrieval models have long dominated the information retrieval space, yet a new method, Latent Terms, challenges their supremacy by extracting sparse retrieval capabilities from them. This approach demonstrates that models trained for dense retrieval, regardless of being single- or multi-vector, inherently learn representations capable of sparse feature decomposition. The key contribution: Latent Terms manages this feat without needing retrieval-specific adjustments.

Breaking Down Dense Models

By employing Sparse Autoencoders, the process reveals a latent vocabulary with nearly Zipfian distribution, making it directly applicable for traditional sparse retrieval scoring such as BM25. It's a significant breakthrough because it bypasses the need for learned expansion objectives or sparse retrieval supervision. The method can be applied to any dense retriever, transforming how we view their inner workings.

Why does this matter? Dense models often obscure the potential of the underlying data. Latent Terms suggests these models are more expressive and contain indexable structure that their default scoring functions don't fully exploit. Essentially, we might have been underestimating what dense retrievers can accomplish all along.

Performance and Implications

Latent Terms not only matches but sometimes surpasses single-vector scoring methods from its base model, even outperforming some SPLADE variants. Impressively, it excels on the LIMIT task, which is specifically designed to highlight single-vector retrieval's limitations. This challenges the assumption that dense models are inherently limited in sparse tasks without significant modifications.

Crucially, this method opens up possibilities for more efficient and interpretable retrieval systems. Why rely solely on dense methods when they can offer more with minimal changes? This adds a new layer to the ongoing debate about the future of retrieval systems and their architectures.

Reflecting on Retrieval Future

The key finding here's that dense models aren't just black boxes. They hold potential for classical retrieval methods when approached with the right tools. Could this mean a shift back towards sparse methods, now enhanced by dense model training? It's a question worth pondering, especially for researchers and developers looking for cost-effective and high-performing retrieval solutions.

, Latent Terms isn't just a technical achievement. It's a reminder that sometimes, the solutions we seek aren't in completely new models but in reevaluating and unlocking the power within existing ones. As the retrieval landscape evolves, integrating sparse and dense methodologies might just be the next frontier.