Rethinking Loss Functions: The Rise of Harmonic Loss

Cross-entropy loss has been the go-to for training deep neural networks. However, it's not without its flaws. Issues like unbounded weight growth and inefficient training dynamics make it less than ideal in certain scenarios. Enter harmonic loss, a distance-based method drawing from Euclidean geometry, promising better interpretability and addressing problems like delayed generalization.

Exploring New Frontiers

The paper's key contribution: expanding harmonic loss beyond just Euclidean distance. The study methodically evaluates various distance metrics, aiming to make harmonic loss a more versatile option. On vision tasks, cosine distances stood out, enhancing accuracy while also lowering carbon emissions. interpretability, the Bray-Curtis and Mahalanobis distances added further depth, though at different efficiency costs.

In language models, the story is similar. Cosine-based harmonic losses brought improvements in gradient stability and strengthened representation structures. Crucially, they also reduced emissions compared to both cross-entropy and Euclidean approaches. If sustainability is a focus, this matters. Who wouldn't want a greener training process?

The Impact on Sustainable AI

Why should we care? The ablation study reveals that by choosing the right distance metric, we can make neural network training not only more interpretable but also more eco-friendly. As AI deployment scales, the emissions from training large models can't be ignored. This research is a step toward more sustainable AI.

But there's still work to be done. The study falls short of a full-fledged evaluation of computational efficiency, which leaves questions open. Can harmonic loss truly replace cross-entropy across the board? Time will tell, but the potential is undeniable.

Code and data are available at: https://anonymous.4open.science/r/rethinking-harmonic-loss-5BAB/.