OARelatedWork: A New Dataset Challenges LLMs with Multi-Document Summarization

Research in natural language processing reaches new heights with the introduction of OARelatedWork, a dataset poised to transform how we generate related work sections from academic papers. Comprising 94,450 papers and over 5.8 million referenced documents, this dataset marks the first large-scale attempt to synthesize entire related work sections from comprehensive full texts rather than mere abstracts.

The Challenge of Full-Text Synthesis

OARelatedWork shifts the traditional focus from abstract summaries to full-text synthesis, an area where even the most advanced large language models face hurdles. The paper's key contribution is its benchmarking of various models, revealing significant drops in True rate when models like GPT-4o-mini transition from abstract-based to full-text contexts. Specifically, the True rate tumbles from 92.9% to 83.8%. This decline underscores the complexity of interpreting and summarizing vast quantities of data.

Why does this matter? The ability of LLMs to accurately synthesize information from extensive documents could redefine how researchers compose literature reviews. But with such a steep drop in performance, are our current models genuinely equipped for this task?

Human vs. Machine: A Surprising Upset

OARelatedWork also offers intriguing insights into human writing behavior. By evaluating 40 papers and 408 factual statements, the dataset reveals that human authors frequently introduce claims not directly grounded in source texts. This behavior gives advanced LLMs an unexpected edge, they surpass human baselines in evidence-grounded factuality. This builds on prior work from the domain, suggesting that while machines may struggle with context, they excel in sticking to the facts.

The ablation study reveals fascinating discrepancies between human and machine performance. Could we soon see a future where machines not only assist in drafting academic papers but also enhance their factual accuracy?

Rethinking Evaluation Metrics

Finally, the creators of OARelatedWork recognize that standard reference-based metrics fall short in evaluating long-form, structured outputs like related work sections. They've introduced a strong statement-level evaluation framework to fill this critical gap. This move challenges the community to rethink how we assess the capabilities of LLMs in complex tasks.

In an era where AI is set to disrupt academic writing, OARelatedWork is a bold step towards more comprehensive and accurate text generation tools. As researchers, we should watch closely as this dataset drives innovation and possibly shifts paradigms in academic writing.