LLM-Wikirace: Unveiling the Shortcomings of Today's AI Planning

LLM-Wikirace is a new benchmark testing large language models (LLMs) on planning, reasoning, and world knowledge. The task is straightforward: navigate from one Wikipedia page to another using hyperlinks, evaluating both look-ahead planning and the understanding of real-world connections.

Models in the Spotlight

The benchmark isn't just a test. It's a challenge thrown at some of the most advanced LLMs like Gemini-3, GPT-5, and Claude Opus 4.5. On simpler tasks, these models show impressive results, even surpassing human capabilities. But when the difficulty ramps up, their performance plummets. Gemini-3, the best performer, only succeeds 23% of the time on hard tasks. That's a glaring gap in their supposed superhuman prowess.

Where Models Fall Short

Strip away the marketing and you get a clear picture: these models struggle with complex planning and long-term reasoning. World knowledge is essential, but once you hit a certain threshold, it's no longer enough. The architecture matters more than the parameter count here. Even top models frequently fall into loops when attempting to replan after a failure.

Why does this matter? Because it underscores a critical limitation in current AI capabilities. If models can't efficiently navigate a structured environment like Wikipedia, how can we expect them to handle real-world complexities?

The Path Forward

LLM-Wikirace offers more than a glimpse into the state of AI. It provides a public arena for developers to push the boundaries of what LLMs can do. But here's the catch: until models can successfully tackle these planning hurdles, their real-world applications will remain limited.

So, what's the takeaway? The reality is, despite advances, LLMs have much to prove in reasoning and planning. The numbers tell a different story from the hype. Can these models evolve to meet the demands of increasingly complex tasks? The answer will define the next era of AI development.