Why Regression Models Need Better Measures Than RMSE

Tabular foundation models like TabPFN and TabICL are shaking up the field, already offering full predictive distributions. Yet, how do we measure their efficacy? Unfortunately, most regression benchmarks rely on point estimate metrics like RMSE and R2. While these are handy, they often miss the mark, especially the tails of the distribution.

The Problem with Traditional Metrics

Why does this matter? In fields like finance and clinical research, the stakes couldn't be higher. Here, asymmetric risk profiles are common, meaning you can't afford to ignore performance in those distribution tails. Relying solely on aggregate measures like RMSE can be downright dangerous, potentially hiding critical model shortcomings.

ScoringBench: A New Approach

Enter ScoringBench, an open benchmark that's looking to change the game. This tool computes a comprehensive suite of proper scoring rules such as CRPS, CRLS, Interval Score, Energy Score, and Brier Score, alongside the standard point metrics. It's like having a full suite of diagnostic tools rather than just a thermometer.

And the results are telling. Evaluating the fine-tuned realTabPFNv2.5 and TabICL against the untuned realTabPFNv2.5, we see that model rankings shift based on the scoring rule used. If that doesn't make you question your current evaluation approach, what will?

Why Should You Care?

Ask yourself: do your models need to anticipate extreme events? If so, the choice of evaluation metric is as essential as the data itself. The real question is, why aren't we more discerning about which metrics to use?

For any domain where missing an outlier could mean financial loss or worse, a regulatory mishap, ScoringBench offers a much-needed reality check. It's a wake-up call for everyone who thought RMSE was enough. And let's face it, in today's data-driven world, who can afford complacency?

ScoringBench is here, atthis GitHub link, and it's ready to hold your models accountable. A live leaderboard is available too, updated via git pull requests, ensuring that transparency and reproducibility aren't just buzzwords.