Why Multimodal Models Need a Positional Revolution

Multimodal Large Language Models (MLLMs) are all the rage for video understanding, but there's a big issue lurking. They're stumbling on multi-video inputs. It's not just about processing power or algorithms. the order of video inputs can twist the quality of the summaries.

Benchmark Breakdown

Let's talk benchmarks. Researchers crafted a test from ActivityNet and News videos, focusing on categories like Cooking, Domestic, Leisure, and News. The twist? They used two- and four-video inputs to see how position messes with the output. Nine different MLLMs, both open-source and proprietary, were put to the test.

The results? Wild. Position effects were all over the place, depending on the domain and the model. Some models showed a minor signed directional bias, but the middle positions often dropped the ball. More visual or generative resources didn't magically fix it. This isn't just a tech hiccup, it's a serious flaw.

The Positional Puzzle

Here's the kicker: Even when the content doesn't change, the slot it occupies does. Why are these sophisticated models so sensitive to position? Are they built on a house of cards? It seems the current systems can't handle the pressure of multi-video inputs without a massive overhaul in their input protocols.

Some attempts at prompt-level mitigation were made, but these are band-aid solutions at best. The takeaway? We need better, order-invariant multimodal systems. The labs are scrambling to figure this out. But until they do, the reliability of MLLMs for multi-video tasks remains questionable.

Implications for the Future

This isn't just a nerdy detail. As video content grows, the demand for accurate, reliable MLLMs will skyrocket. These models are supposed to help us understand and summarize vast amounts of video data efficiently. But if they can't even get the order right, where does that leave us?

And just like that, the leaderboard shifts. If one lab nails this, they could redefine video AI. But until then, users are left wondering if they can really trust these models' outputs. Position should complement the content, not define it. Let's hope the next wave of MLLMs gets it right.