Video Games: The New Frontier for Vision-Language Models

In the labyrinth of artificial intelligence, vision-language models (VLMs) have long been celebrated for their prowess in handling complex challenges, such as coding and math, that often flummox their human counterparts. Yet, tasks that humans breeze through, like perception, spatial navigation, and memory management, these models stumble. Enter VideoGameBench, a new benchmark designed to test these human-like skills in AI, using a surprisingly intuitive playground: video games from the 1990s.

The Challenge of Intuition

Video games, especially those designed in the '90s, are built on innate human biases and learning capabilities. They make for an excellent testbed to evaluate AI's performance on tasks that humans find natural. VideoGameBench does exactly this, challenging VLMs to play 10 popular retro video games without the usual crutches of game-specific guidance and auxiliary data. Models receive raw visual input and a high-level outline of objectives and controls. It's a departure from typical setups, intended to assess how well these models can generalize to new environments without a guided hand. And, to keep the playing field honest, three of the games remain undisclosed, pushing models to adapt creatively.

Frontier Models Meet Their Match

Let's apply some rigor here. The results are telling: the new models, Gemini 2.5 Pro and Claude 3.7 Sonnet, have barely scratched the surface, completing a meager 0.48% of VideoGameBench and an only slightly better 1.6% of a more forgiving setup, VideoGameBench Lite. In this variant, the game halts to accommodate the model's processing time, highlighting a critical issue: inference latency.

What they're not telling you is the extent of this limitation. Real-time decision-making remains a significant hurdle for VLMs, which seem to buckle under the pressure of rapid-fire action required by these games. It begs the question: if these models can't handle the relatively straightforward task of playing a retro video game, what hope do they've in more complex, real-world scenarios that demand similar skills?

Paving the Way Forward

VideoGameBench is more than a mere benchmark. It's a challenge to the AI community to enhance these models' capabilities in ways that align more closely with human cognitive functions. The goal is clear: to spur advancements that could allow VLMs to navigate tasks that require not just raw computational power, but genuine understanding and quick adaptation.

Color me skeptical, but unless the community takes this challenge seriously, VLMs might remain impressive on paper yet struggle in practical applications. The introduction of VideoGameBench and its Lite counterpart should serve as a clarion call for researchers to address these shortcomings. It's not enough to excel in static, pre-defined tasks. The real world, and the virtual one of video games, demands more.