Cracking the Code: Enhancing LLMs in Multi-Turn Medical Diagnosis

Large language models (LLMs) have impressed with their diagnostic accuracy when all clinical data is thrown at them simultaneously. But the real challenge? Handling information as a real doctor would: incrementally, over several turns. Enter the Medical Incremental N-Turn Benchmark (MINT), a novel framework pushing LLMs into uncharted territory.

MINT Unveiled

MINT isn't just another test. It's a rigorous benchmark with 1,035 medical cases, dissected into controlled, information-rich turns. It's designed to reveal how LLMs behave when forced to accumulate evidence over multiple interactions. This matters because a single-turn approach is rarely how diagnoses unfold in the real world.

Through MINT, researchers uncovered three behavioral quirks of LLMs that throw a wrench in their diagnostic prowess. First, the eagerness to answer without sufficient evidence. Over 55% of responses come prematurely, within just the first two turns. Why is this a problem? Because it exposes an inherent impatience that undermines the very reasoning process we aim to emulate.

Self-Correction and Strong Lures

Interestingly, LLMs show a latent self-correction ability. They revise incorrect answers to correct ones at a rate up to 10.6 times higher than the reverse. This could be a boon if harnessed wisely, but it’s often thwarted by premature decisions.

Then there’s the issue of “strong lures.” Laboratory results and other critical data prompt LLMs to jump to conclusions, even when instructed to pause for more evidence. It’s a problem of allure versus patience, and it costs accuracy.

The Path Forward

So, what’s the fix? Crucially, deferring key diagnostic questions to later stages can slash premature responses, boosting accuracy by an impressive 62.6%. Additionally, strategically reserving critical clinical evidence for later turns prevents a staggering 23.3% drop in accuracy that premature decisions cause.

Why should we care? Because as LLMs become more integrated into healthcare, their reliability is non-negotiable. MINT provides a roadmap to refine these models, ensuring they offer value in a real clinical setting. But the question remains: will developers heed these insights and adjust their models accordingly?

The paper’s key contribution: it’s not just a critique but a call to action. For LLMs to truly aid in medical diagnostics, they need to mimic the careful, evidence-driven approach of human clinicians. Anything less, and they risk becoming just another over-hyped tech experiment gone wrong.