ToolSense: Exposing the Retrieval Flaws of Large Language Models

Large language models (LLMs) may be the darlings of AI labs, but they're hitting a serious snag retrieving tools from expansive catalogs. The traditional embedding-based retrieval systems just aren't cutting it. These systems rely on compact encoders which often fail to grasp the nuances of specialized tool semantics.

Parametric Retrieval: A New Hope?

Enter parametric tool retrieval. It aims to solve this bottleneck by encoding each tool as a virtual token in the LLM's vocabulary. The process involves a two-stage fine-tuning: first, memorization, then retrieval-specific fine-tuning (SFT). Sounds promising, right? Indeed, it shows solid performance on the ToolBench retrieval benchmarks. But let's not celebrate just yet.

These benchmarks depend on verbose, fully-specified queries and constrained decoding. So, do they truly test a model's understanding of the tools, or merely its ability to follow a pre-set path?

ToolSense: Lifting the Veil

ToolSense, an open-source framework powered by LLMs, is here to shake things up. By taking any tool catalog as input, it generates three benchmarks: a Realistic Retrieval Benchmark (RRB) with queries across three ambiguity tiers, and both MCQ and QA probing benchmarks.

When applied to ToolBench, with approximately 47,000 tools, ToolSense unearthed a startling knowledge-retrieval dissociation. On the RRB queries, many training configurations saw their scores collapse by 50-64 percentage points compared to their performance on ToolBench's fully-specified queries. This plummets their performance below even the baseline embedding models.

Why This Matters

Here's the kicker: despite strong retrieval scores, some models performed near-randomly on factual probes. What does that tell us? It suggests these models are parroting rather than understanding. Slapping a model on a GPU rental isn't a convergence thesis if it can't even differentiate between tool names and functions.

So, why should you care? If the AI can hold a wallet, who writes the risk model? These gaps highlight the need for more rigorous diagnostic tools in AI development. The intersection is real. Ninety percent of the projects aren't. Yet, with frameworks like ToolSense, we're closer to discerning which projects will stand the test of real-world application.

The ToolSense framework and diagnostic benchmarks are freely available at https://github.com/SAP/toolsense. It's about time we stop taking high benchmark scores at face value and start questioning what lies beneath.