Unpacking Explainability in Machine Learning: The Real Issue at Decision Boundaries

Explainability in machine learning isn't just a desirable feature. In many critical applications, it's a regulatory necessity. Yet, methods like LIME and SHAP often face criticism for their lack of stability near decision boundaries. Is this critique fair? Not exactly. The disruption stems from high forecast uncertainty at these boundaries, not from flaws in the methods themselves.

Rethinking Explanations

Let's consider the sequence of questions we're asking. Nonlinear models can be highly predictive in some regions but nearly useless in others. The first step should be determining if a forecast actually exists with low enough uncertainty to be practical. If it does, then and only then should a local linear explanation be pursued. Explanatory instability reduces when the forecast is sound. In cases where the forecast isn't reliable, simpler models like logistic regression should be used instead.

The Illusion of Explainability

Models that claim to be explainable everywhere, like ReLU networks or piecewise linear models, offer only an illusion. Why? The forecast uncertainty at segment boundaries is simply too high to be meaningful. If the forecast isn't usable, any explanation becomes pointless. It's like drawing a map for a territory that doesn't exist. The paper's key contribution: we need to focus first on the quality of the forecast, then on explaining it.

Why It Matters

There's a key insight here for machine learning practitioners and policymakers alike. Misinterpretations around stability could lead to misplaced trust in models that aren't providing reliable predictions. But who benefits from this misunderstanding? Certainly not the end-users making critical decisions based on these predictions.

For developers, this means revisiting the sequence of our methodologies. For regulators, it's a call to ensure that explainability isn't just a checkbox but a meaningful aspect of model evaluation. When we talk about machine learning explainability, let's get the sequence right.