Video MLLMs: Safety Monitors or False Alarms?

Between spotting danger and an accident actually happening, there's often a essential window where we can still act. Multimodal large language models (MLLMs) with video capabilities are marketed as possible 24/7 safety watchdogs, primed to issue warnings during this critical period. But are they genuinely up to the task?

The PaSBench-Video Challenge

A new benchmark, PaSBench-Video, throws down the gauntlet. With 740 videos spread across driving, healthcare, daily life, and industrial production, it tests these models in 481 risk and 259 no-risk settings. The challenge: identify risk onset and accident boundaries with precision. Sounds straightforward, right? Yet, none of the 13 MLLMs tested managed to exceed a 20% success rate on the toughest metric.

That raises a significant question. If these models are meant to be our safety net, why are they missing the mark so dramatically? What's more, their performance fluctuates wildly depending on the domain. While recognizing odd incidents in daily life, they flounder with driving videos, often flagging both routine and risky scenes indiscriminately. The pitch deck says one thing. The product says another.

The False-Positive Conundrum

A major sticking point is the balance between detecting true risks and avoiding false alarms. PaSBench-Video illustrates that more detections often mean more false positives, with a Pearson correlation of 0.64. In layman's terms, if you push these models for better recall, you're likely to end up with a slew of unnecessary warnings. And in life-or-death domains like driving, that's not just annoying, it's potentially dangerous.

So, what's the takeaway? The metrics here tell an intriguing story. The founder story is interesting. The metrics are more interesting. Current MLLMs don't seem to grasp the nuances of emerging threats, relying instead on broad activity cues. This isn't just a technical hiccup. it's a fundamental flaw that needs addressing before we entrust these systems with real-world safety.

Real-World Implications

For startups developing these models, it's a wake-up call. The grind is just beginning, and the real story is about refining these systems to handle the complexities of varied environments. Fundraising isn't traction, and the focus should shift to reducing false positives while maintaining high recall in critical applications.

In the trenches of AI safety, one has to wonder: Are we prematurely leaning on technology not quite ready for prime time? What matters is whether anyone's actually using this, not just if it can theoretically work. Until MLLMs can reliably differentiate between safe and unsafe scenarios, their role as safety monitors remains questionable at best.