Rethinking Automation in Machine Learning: The Case for Feature Preprocessing

In the ever-expanding world of machine learning, classical models like linear and tree-based ones remain steadfast in their utility. However, these models possess a glaring sensitivity to data distribution, making feature preprocessing indispensable for ensuring quality outcomes. Transforming features between distributions is no small feat, and the manual construction of a preprocessing pipeline can be a daunting task for data scientists.

The Challenge of Automation

Automating feature preprocessing for tabular data, or Auto-FP, presents a tantalizing opportunity to speed up this process. Yet, this endeavor is fraught with complexity. The search space is vast, and a brute-force approach proves to be both impractical and costly. Interestingly, researchers have found that Auto-FP can be framed as a problem of hyperparameter optimization (HPO) or neural architecture search (NAS). This opens the door to applying various algorithms to tackle the challenge.

A comprehensive evaluation conducted on 45 public machine learning datasets examined 15 algorithms. Evolution-based algorithms emerged with the best average ranking, but here's the kicker: random search, often underestimated, stood out as a surprisingly strong baseline. This finding challenges the conventional wisdom that surrogate-model-based and bandit-based search algorithms excel across the board.

Why Random Search Holds Its Ground

One might wonder, why does random search perform so well in this setting? The deeper question lies in the nature of the search space. Many advanced algorithms bring unnecessary complexity to a problem that, at its core, may not require such intricate solutions. This suggests that we need a more nuanced understanding of the problems we aim to solve with automation.

The findings also highlight opportunities to refine our approach. Bottleneck analysis reveals points of improvement for existing algorithms, suggesting that we may need to rethink our strategies rather than rely on the novelty of advanced methods alone.

Beyond the Algorithms: The Broader Implications

Evaluating Auto-FP in the context of AutoML tools exposes limitations in popular solutions. These tools often fall short of handling the intricacies of feature preprocessing effectively, which raises a critical question: Are we relying too heavily on automation without truly understanding the underlying challenges?

This study, the first of its kind, ought to inspire researchers to develop algorithms uniquely tailored for Auto-FP. The field stands on the brink of transformation, but it requires a careful balance between novel approaches and tried-and-true methods.

As we look to the future, we must not shy away from questioning the status quo. Are we too quick to assume that complexity equals performance? This exploration into Auto-FP may well be the catalyst for a more thoughtful approach to automation in machine learning.