Why LLMs Aren't Ready for Prime Time in Decision-Making

Large language models (LLMs) have become the juggernauts of AI, making waves in everything from chatbots to text generation. But helping decision-making agents navigate the tricky waters of exploration-exploitation tradeoffs, they might not be ready for the big leagues just yet.

Exploration vs. Exploitation: A Classic Dilemma

If you've ever trained a model, you know the eternal struggle between exploring new possibilities and exploiting known options. Think of it this way: should a model keep trying new paths, or stick to what it knows works? This isn't just an academic curiosity. It's a practical problem in fields like advertising, healthcare, and beyond.

Recent studies have tested LLMs to see if they can independently tackle exploration and exploitation tasks in various bandit scenarios. The results? Mixed, at best. While reasoning models showed a glimmer of promise, they're often too slow or computationally expensive to be practical. So, where does that leave us?

The Cost of Complexity

Here's the thing: even when LLMs are enhanced with tools like in-context summarization, they struggle. For medium-difficulty tasks, these enhancements might offer some performance boost. But ultimately, even the most advanced LLMs are outperformed by simple linear regression. Yes, you heard right. Linear regression, the classic workhorse of statistics, still edges out these massive models in non-linear settings.

This raises an important question: Why are we pouring resources into these complex models when simpler ones get the job done just as well, if not better? It's a sobering thought for anyone betting on AI to solve all our problems.

Navigating Large Action Spaces

Despite these shortcomings, it's not all bad news. LLMs do have an edge exploring large action spaces filled with inherent semantics. They're quite adept at suggesting candidates to explore, which could be invaluable in areas like drug discovery or recommendation systems, where the space of possibilities is vast.

Here's why this matters for everyone, not just researchers: As we push the boundaries of AI, understanding where these models excel and where they falter helps us make smarter decisions about when and how to deploy them. Blindly applying LLMs to every problem might not just be inefficient, it could lead us down the wrong path entirely.

In the end, the analogy I keep coming back to is a Swiss Army knife. LLMs are versatile and capable, but they're not the perfect tool for every job. Understanding their limitations is just as important as celebrating their successes.