LLMs' Explanations: Insightful or Misleading?

Just when you thought large language models (LLMs) were getting smarter, a new study shows they're still stumbling over their own explanations. When LLMs make predictions and try to explain them, there's a wild pattern. Correct predictions usually come with explanations that point to the right clues in the text. But when they get it wrong, boy do they get it wrong. The explanations suddenly reference all the wrong signals.

The Study

Researchers dove into three datasets: WIKIONTOLOGY, AG NEWS, and IMDB. They compared LLM-generated explanations with feature importance signals from straightforward linear models like logistic regression and linear SVM. Turns out, there's a consistent pattern that the team calls 'support-contra asymmetry.' Explanations that align with correct predictions highlight more supportive evidence. But when predictions miss the mark, the explanations are packed with contradicting evidence.

The Curious Case of Misleading Explanations

So what gives? Why are LLMs referencing misleading cues when they make a mistake? Are they trying too hard to justify the unjustifiable? LLMs, explanations are supposed to help us trust the model's decisions. But if they're pointing in the wrong direction, that trust gets shaky.

This isn't just an academic exercise. It has real implications for how we use these models in critical tasks. After all, if your AI assistant is confidently wrong, you'd better know why.

What's Next for LLMs?

The labs are scrambling to figure out how to fix this. With the current trajectory, relying on LLMs for tasks requiring high stakes decision-making could be risky. There's a massive gap between what the models predict and what their explanations justify. It's like a scenic route that leads nowhere.

So here's the hot take: Developers need to tighten the screws on how LLMs generate explanations. It's not enough for them to be right. They need to know why they're right and when they're wrong, avoid adding salt to the wound.

And just like that, the leaderboard shifts. The next-gen of LLMs has to do better. They need explanations that don't just sound convincing but actually reflect the true nature of the task at hand.