The Imperfect Science of Citation: Large Language Models Under Scrutiny

As the reliance on large language models (LLMs) with web search capabilities grows in scientific publishing, a recent investigation unveils a troubling truth: these models continue to produce citations riddled with errors. With a benchmark set across 931 papers from diverse scientific domains, the study shines a light on the limitations of current AI tools, even when equipped with search functionalities.

Dissecting the Accuracy

The benchmark involves three advanced models, GPT-5, Claude Sonnet-4.6, and Gemini-3 Flash. These models were tested for their ability to accurately generate BibTeX entries across nine specific fields, resulting in an overall accuracy of 83.6%. Yet, this figure is misleading. Only half of the citations, a mere 50.9%, were entirely correct. Such a discrepancy raises questions about the reliance on these technologies in scholarly work.

What becomes evident is the models' dependence on parametric memory. For popular papers, the accuracy was higher, but it plummeted by 27.7 percentage points when tasked with citing more obscure, recent works. This suggests that despite the availability of search capabilities, these LLMs lean heavily on pre-existing data, failing when new information comes into play.

Understanding the Errors

The study categorizes errors into two main types: wholesale entry substitution, where identity fields collectively fail, and isolated field errors. Such findings stress the necessity for better citation management within AI systems. In an environment where precision is critical, these errors are far from trivial.

Is it acceptable for half of citations to be inaccurate in academic publishing? The answer should be a resounding no. The integrity of scientific documentation depends on precise and verifiable citations. This isn't just about technology, it's about maintaining the trust and validity of scholarly communication.

Mitigation and

Enter clibib, an open-source tool aiming to address these shortcomings. By integrating BibTeX retrieval from the Zotero Translation Server with a CrossRef fallback, clibib offers a mitigation strategy. In a two-stage process, where entries are revised against authoritative records, accuracy notably improves by 8 percentage points, reaching 91.5%. Fully correct entries rise to 78.3%, with regressions nearly negligible at 0.8%.

This improvement underscores a critical point: the architecture of integration matters more than the model's inherent capability. Separating the search process from revision yields better outcomes, showcasing a path forward for other tools facing similar challenges.

In the ongoing quest for progress, should the academic community settle for anything less than utmost accuracy? The pressing need for a reliable solution is clear. In the immediate future, tools like clibib might bridge this gap, but the fundamental question remains: how do we ensure that technology keeps pace with the demands of academia?