The Pitfalls of Logistic Regression: Why Rotation-Invariance Isn't Always Your Friend

Logistic regression is often hailed as a cornerstone of machine learning, yet its simplicity can be deceptive. A recurring issue emerges in scenarios where models seek to learn noise-free soft targets. Particularly, when the dataset is over-constrained, meaning the number of samples outstrips the input dimensions, things get tricky.

The Problem with Hard Labels

In such setups, learning the underlying weight vector, denoted asw^⋆, becomes essential. If successful, one can achieve what's known as the Bayes risk, the lowest possible predictive error. However, here's the rub: when examples are tagged with hard labels, discrete values derived from the underlying probability distribution, rotation-invariant algorithms stumble.

These algorithms, like the ubiquitous gradient descent on logistic loss, end up with an excess risk that's inverse to the number of samples, specifically,Ω((d-1)/n), wheredis the dimensionality. That's a problem when you're aiming for optimality. Color me skeptical, but relying on rotation-invariance here seems an oversight.

Why Rotation-Invariance Fails

What exactly is the flaw with rotation-invariant algorithms? Simply put, they lack the nuance needed to handle s-sparse targets, scenarios where only a small subset of features actually matters. In contrast, algorithms that aren't bound by rotation invariance, employing strategies like reparameterizing weights into products, offer a superior risk ofO(s log(d)/n).

I've seen this pattern before: a model's elegance often comes at the cost of pragmatism. It's a classic case of overfitting to a theoretical ideal rather than addressing the messiness of real-world data. Who benefits from a model that works beautifully in theory but falters when confronted with actual data?

Looking Beyond Elegance

The lesson here isn't just technical. It's a reminder to scrutinize the underlying assumptions of our models. Rotation-invariance might sound appealing, yet in the context of logistic regression with sparse data, it's akin to wearing blinders. The claim doesn't survive scrutiny when faced with the nuances of s-sparse targets.

the allure of sticking with tried-and-true algorithms is strong. But innovation often requires stepping outside comfort zones, even if it means abandoning the elegance of rotation-invariance for more tailored, effective solutions.

As machine learning continues to evolve, it becomes essential for practitioners to question the orthodoxies of model design. After all, why settle for suboptimal when the tools for precision are within reach?