LLMs: Can They Really Learn Experimentally?

Recent discourse in the AI community questions whether large language models (LLMs) like Claude Sonnet can genuinely learn from context in scientific settings. Specifically, their ability to adapt based on experimental feedback has been under scrutiny. A comprehensive study offers some answers, experimenting with iterative perturbation discovery.

Experimenting with Feedback

The study conducted 800 independent experiments using a technique known as Cell Painting for high-content screening. Two different approaches were compared: an LLM that iteratively updates its hypotheses using feedback, and a baseline model relying solely on its pretraining knowledge.

The results are compelling. Access to feedback led to a 53.4% increase in discovery rate per feature. This is statistically significant with a p-value of 0.003. But what happens if the feedback is random? The study introduced a control with random hit/miss labels, and the performance gain vanished. This validates that the improvement is genuinely due to structured feedback.

Model Capabilities and Learning

But here's the kicker: it's not just about having access to feedback. Model capability plays a turning point role. Upgrading the model from Claude Sonnet 4.5 to 4.6 significantly reduced gene hallucination rates from about 33%-45% to a mere 3%-9%. This upgrade turned a non-significant in-context learning effect into a notable positive outcome.

Why should you care? This finding highlights that effective learning from feedback isn't just a given. it needs a model to be on a capability threshold. So, are we overestimating the prowess of LLMs in real-world applications? It seems there's room for skepticism about their current capabilities.

A Call for Realism

Developers should note this breaking change in expectations. While LLMs have potential, they aren't yet the infallible learners some might assume. The specification is as follows: real-time feedback is valuable, but only when the model is technologically advanced enough to handle it.

For those in AI research and development, this study is a reminder. An LLM’s learning prowess depends heavily on both the quality of input and the sophistication of the model. As AI continues to evolve, this serves as a benchmark for future improvements and assessments.