Rethinking AI Tool-Use: The Amazing Agent Race Challenges Linear Benchmarks

As AI models continue to evolve, the benchmarks we use to evaluate them must also advance. The Amazing Agent Race (AAR) is a new benchmark designed to push the limits of AI agents beyond the linear tool-use that dominates existing tests. The paper, published in Japanese, reveals essential insights into how AI navigates complex tasks.

Challenging the Status Quo

Existing benchmarks are overwhelmingly straightforward, often involving simple chains of 2 to 5 steps. But AAR offers a different approach. It introduces directed acyclic graph puzzles, also known as 'legs', which require agents to handle fork-merge tool chains. This complexity forces AI to navigate Wikipedia and execute multi-step processes to reach verifiable answers.

AAR's introduction of these DAG puzzles is significant. With 1,400 instances split into sequential and compositional variants, it provides a comprehensive test across four difficulty levels. Notably, live-API validation ensures that the tasks remain grounded in real-world applications.

Performance Metrics: Revealing AI's Weaknesses

The benchmark includes three complementary metrics: finish-line accuracy, pit-stop visit rate, and roadblock completion rate. These metrics diagnose different areas of failure, navigation, tool-use, and arithmetic errors. Compare these numbers side by side with existing tests, and the gaps become apparent.

Results from testing three agent frameworks show a top accuracy of only 37.2%. Navigation errors, notably, account for 27 to 52% of trials. In contrast, tool-use errors are below 17%, suggesting that AI struggles more with finding the right information than with using tools effectively.

Architecture Over Scale: A Surprising Insight

Interestingly, the data shows that architecture matters as much as model scale. For instance, Claude Code matches Codex CLI in performance despite using six times fewer tokens. This raises a critical question: Are we focusing too much on scaling up when architectural innovation might yield better results?

What the English-language press missed is that the compositional structure of AAR reveals a significant blind spot in current AI evaluation. Linear benchmarks simply don't capture the complexity of real-world tasks that AAR does. This could be the wake-up call needed to shift focus from scaling models to refining their design.

For those interested, the project page offers further insights into AAR's mechanics and outcomes. It’s a reminder that as AI marches forward, our benchmarks must not only keep pace but also drive innovation.