Unlocking Data's True Value: The Surprising Reality of Dataset Appraisal

Neural scaling laws and the Vendi Score have been hailed as new ways to appraise datasets' value. But do they really live up to the hype? Recent findings suggest a more nuanced picture.

The Trouble with Common Metrics

Everyone loves a good metric, right? Neural scaling laws evaluate data through sheer size, while the Vendi Score, with its quantum entropy roots, promises to measure dataset value. Both have been found to be submodular, a property that simplifies optimization. However, there's a twist.

The Vendi Score is actually part of a broader class of submodular objectives dubbed matrix spectral functions. This family includes determinantal point processes (DPPs) and others, opening the door to a range of appraisal methods.

Speed and Efficiency in Data Appraisal

Here's where the magic happens: researchers developed an update method that sidesteps repeated eigendecompositions, essential for greedy optimization. This innovation cuts the time it takes to evaluate $m$-dimensional embeddings by an impressive factor of $O(m)$ compared to traditional methods. The result? A jaw-dropping average speedup of about 35,000 times. Now, optimizing the Vendi Score on massive datasets like ImageNet-1K is within reach.

But does faster mean better? The research says not necessarily. When put to the test, facility location objectives consistently outperformed others, even the much-touted Vendi Score. It seems that in the real world, the size and balance of your dataset aren't the only things that count.

The Real Story of Dataset Value

Here's the kicker: randomly selected, fixed-size subsets, whether constrained by class balance or not, showed remarkable consistency in both appraisal scores and test performance. This finding challenges the notion that bigger or precisely balanced datasets always translate to better outcomes.

So, if size and balance aren't the sole indicators of value, what's? The truth is, performance varies significantly, from stellar to subpar, even when controlling for these factors. The gap between theoretical measures and practical outcomes is enormous.

Does this mean it's time to ditch the Vendi Score and neural scaling laws altogether? Not quite. While they're not the holy grail some might hope for, they still offer insight, especially within moderate score ranges. But we can't ignore the evidence: facility location stands out as a more reliable predictor of dataset value.

In the end, it's not merely about the tools we use but how we interpret and apply them. The real story of data appraisal is one of complexity and context. Will we continue to rely on traditional metrics, or will we embrace the nuances that facility location and its ilk bring to the table?