MiraBench: Rethinking Reliability in Robotic World Models

Robotic learning has long relied on world models that simulate potential futures based on actions. But can these models be trusted to make accurate predictions? Enter MiraBench, a new benchmark that shifts the focus from visual fidelity to action-conditioned reliability, the critical aspect for truly effective robotic models.

The Problem with Visual Fidelity

Historically, benchmarks have prioritized how realistic the generated environments look. However, MiraBench underscores a key flaw: visual fidelity doesn't equate to accurate action prediction. In robotics, a model's value is determined by its ability to predict results based on specific actions. Can the robot execute the task it's been programmed for? That's the real question.

MiraBench's Three-Tiered Approach

MiraBench breaks down action-conditioned reliability into three layers. First, Physics Adherence ensures the model maintains basic physical laws, even without direct reference points. Second, Action-Following Fidelity checks if predictions align with the tasks the robot's supposed to perform. Third, Optimism Bias Detection identifies where models incorrectly predict success, even when failure should occur. This last aspect is especially vital, as it exposes tendencies within models to overestimate their own capabilities.

The benchmark's creation involved over 16,000 human-annotated judgments, spanning several tasks and failure categories. That's not a small feat. It evaluated 12 diverse model configurations, including vector-conditioned and text-conditioned generative models, as well as open-weight and closed-source systems.

The Findings: A Wake-Up Call

MiraBench's evaluations deliver a clear message. First, visual fidelity is a poor stand-in for actual action fidelity. Secondly, increasing model scale doesn't necessarily improve action following. Lastly, optimism bias is rampant in current systems, a finding that rings alarm bells for anyone relying on these models for robotics.

So why should you care? If you're in the field of robotics, ignoring action-conditioned reliability in favor of appearances could lead to catastrophic failures. Predicting that a robot can do something it actually can't has ramifications, from industrial mishaps to safety hazards in human-robot interactions. We need models that don't just look good but do good.

Ultimately, MiraBench provides a new diagnostic tool for anyone serious about improving robotic world models. It challenges researchers and developers to reassess their priorities and focus on what truly matters: reliable, action-based predictions. Will the industry take heed?