Why You Should Rethink AdamW for Tabular Data Models

If you've ever trained a model, you know the optimizer choice can make or break your training run. For years, AdamW has been the go-to optimizer for supervised learning on tabular data. But a recent benchmark suggests it's time to rethink this automatic choice.

The Rise of Muon

Imagine discovering your trusty optimizer isn't the fastest horse on the track anymore. That's what happened when researchers benchmarked several optimizers on multiple tabular datasets. Surprisingly, the relatively unknown Muon optimizer consistently outperformed the stalwart AdamW. Think of it this way: if you're looking for performance, Muon might just be the ticket you've been waiting for.

Here's why this matters for everyone, not just researchers. As we push for more efficient models, minor improvements can lead to significant real-world impacts. While Muon does come with a training efficiency overhead, if you can afford it, the performance gains could be well worth the trade-off.

Exponential Moving Average: A Simple Trick

In addition to exploring optimizers, the research found that applying an exponential moving average (EMA) to model weights offers a performance boost to AdamW, at least on vanilla MLPs. While not as consistently effective across all model variants, it's a low-hanging fruit that might just give you that extra edge.

So, why aren't we all just switching over to Muon? The analogy I keep coming back to is choosing between a sturdy old car and a shiny new sports car. Muon might be faster and more efficient, but it requires a bit more maintenance compute budget. Not everyone will need or want to make that switch, but it's worth considering if you're chasing top-tier performance.

Rethinking the Status Quo

Honestly, the complacency around optimizer choice in tabular DL models shows how easily we can settle into habits. The performance of Muon is a wake-up call. Are we willing to sacrifice a bit of convenience for a lot of performance?

, the decision ultimately comes down to your specific needs and resources. But one thing's clear: sticking with the familiar might mean missing out on better results. In the fast-evolving field of machine learning, that's a gamble you might not want to take.