Perception Programs: Elevating Multimodal Models Beyond Language Biases

Multimodal language models (MLLMs) are at the heart of AI's future, merging the capabilities of language models with vision tools. Yet, there's a persistent challenge: while these models can access rich visual data, they often underperform because they can't align raw visual inputs with their language processing strengths. This issue has long hindered the models' ability to fully use the capabilities of vision tools like depth and flow sensors.

Perception Programs: A New Approach

Enter Perception Programs, or P². This innovative method tackles the disconnect by translating dense visual data into structured, language-native summaries. The design is training-free and model-agnostic, meaning it doesn't rely on retraining or modifying existing models. It's about presentation, not more computation.

Why should anyone care? P²has shown staggering results. In a series of tests under the BLINK framework, P²increased a base model's accuracy from 41.35% to an impressive 86.47% on multi-view reasoning tasks. That's a leap not just in numbers, but in capability. On relative depth tasks, accuracy jumped from 52.42% to 81.45%. These aren't minor tweaks. they're paradigm shifts.

Why Representation Matters

The AI-AI Venn diagram is getting thicker. This isn't just about improving benchmarks. It's about redefining how we think about AI's ability to interpret the world. P²suggests that the bottleneck isn't in the size of the model or the number of tools it can access, but in how it represents and processes the data those tools provide.

Even smaller models like InternVL3.5-4B and Qwen3VL-4B reaped the benefits, with gains ranging from 15% to 40%. If smaller models can achieve such substantial improvements, what does this mean for future AI development?

Rethinking Model Design

The success of P²challenges the current trend of increasing model size and complexity. Why not focus on smarter data representation? The compute layer needs a payment rail, and P²might just be it. This approach could lead to more efficient models that don't require exorbitant computational resources to deliver high performance.

If agents have wallets, who holds the keys? In essence, P²redefines the keys to AI's capacity for understanding the world. As AI continues to integrate deeper into our lives, ensuring that it sees the world as clearly as it reads language is more important than ever.