Rethinking Data Selection: How DATAPROPHET Could Change MLLMs Training

In the race to train effective multimodal large language models (MLLMs), picking the right supervision data is important. Traditionally, selecting datasets similar to the target benchmark seemed sensible. But what if that's not the best approach?

Challenging Conventional Wisdom

Recent analysis across 14 vision-language datasets, covering seven diverse tasks, reveals a surprising twist. The expectation that task similarity predicts transferability is misleading. Instead, the specific characteristics of each dataset hold more sway over performance. Numbers in context: task similarity isn't the panacea most believe it to be.

Here lies the crux: Can we anticipate how a dataset will influence a benchmark before even starting training? Visualize this: a world where dataset selection isn't a shot in the dark but a calculated decision.

Introducing DATAPROPHET

Enter DATAPROPHET, a groundbreaking metric that combines multimodal perplexity, similarity, and data diversity. It's training-free yet powerful. With a Kendall's tau of 86.0%, DATAPROPHET's rankings of supervision data align closely with post-training performance gains. This isn't just a marginal improvement, it's a breakthrough in data selection strategy.

Why is this important? Because better data selection means better model performance. DATAPROPHET outpaces uniform selection by 6.9%, a state-of-the-art training-based method by 1.4%, and even experimental oracle selection by 0.2%. That's not just incremental, it's transformative.

Why Should We Care?

If data is the lifeblood of MLLMs, then DATAPROPHET is the diagnostic tool we need. It offers a clearer path to data-driven decisions, bypassing the need for extensive training just to test dataset efficacy. One chart, one takeaway: dataset selection needn't be a guessing game.

So, what's the real impact? In a world where data drives progress, tools like DATAPROPHET could redefine how we think about data selection. The trend is clearer when you see it: moving from intuitive guesses to informed choices.

Rhetorical question: Are we ready to rethink our approach and embrace data science's predictive power? The answer might shape the future of model training.