LLMs Face Challenges in Multi-Turn Conversations: A Closer Look at the 'Stick-or-Switch' Framework

Large language models (LLMs) are increasingly integral to our tech-driven world, yet they stumble in multi-turn conversations. The 'stick-or-switch' (SoS) framework sheds light on these challenges, particularly in high-stakes environments like healthcare, where patients and clinicians rely on LLM chatbots.

Conversational Inefficiencies

The SoS framework partitions the question-answer space to evaluate two important behaviors: conviction and flexibility. In simple terms, it examines if models stick to the correct answers or switch to better suggestions. Evaluating 17 LLMs across three clinical benchmarks, the data shows a significant 'conversation tax.' Partitioning an answer-space into multiple presentations drops accuracy and increases incorrect abstentions by up to 30%, in some models hitting a staggering 65%.

Why should we care? Because these inefficiencies could mislead patients seeking medical advice. If models are prone to blind switching, transitioning from abstention to incorrect suggestions at nearly 50% rates, they become less reliable. Notably, larger models aren't immune, sometimes exacerbating these issues by adopting incorrect suggestions more readily.

The Role of Model Scale

It's tempting to think scaling up models solves these problems. The reality is more complex. While larger models might mitigate some conversation inefficiencies, they worsen others, such as a higher tendency to adopt erroneous suggestions from an initial abstention. This discrepancy raises a critical question: Are we prioritizing the wrong benchmarks?

The paper, published in Japanese, reveals how static benchmarks often fail to reflect real-world dialogue dynamics. Compare these numbers side by side with static benchmark results. The differences are stark and suggest that proficiency in static tests doesn't equate to conversational efficacy.

A New Focus for LLMs

Western coverage has largely overlooked this, focusing instead on the raw performance metrics of LLMs. But isn't it time we shifted our focus from static benchmarks to real-world conversational capabilities? After all, the benchmark results speak for themselves. The industry needs to prioritize dialogue efficiency over static accuracy if we want LLMs to be truly useful in real-world applications.

, the 'stick-or-switch' framework offers a important lens through which to view LLM performance. As these models continue to integrate more profoundly into everyday tasks, understanding and addressing their conversational limitations isn't just important, but essential.