LLMs Struggle with Complex Office Tasks: Are They Ready for Prime Time?

Large Language Models (LLMs) have taken the AI world by storm, revolutionizing everything from chatbots to creative writing. However, the intricate task of automating professional-grade productivity software, their skills are being put to the test. Recent evaluations reveal that LLMs may not be as ready for the workplace as many hoped.

The Challenge of Office Automation

Office automation isn't your average AI task. It requires long-horizon planning, precise parameter tweaking, and smooth integration across multiple applications like Word, Excel, and PowerPoint. That's not trivial for AI. To measure their mettle, researchers devised a benchmark using China's National Computer Rank Examination (NCRE), featuring 200 hands-on tasks. Scored on a 100-point rubric with 7,118 criteria, this benchmark offers a rigorous test of LLM capabilities.

LLMs Under the Microscope

Seven leading LLMs took on this challenge, but the results were less than stellar. Single-turn models, which complete tasks in one go, peaked at a mere 36.6% on the rubric. Even when using more sophisticated systems with execution feedback and iterative repair, the scores only climbed to 68.8%. These results starkly contrast with the 95.5% score achieved by human benchmarks, underscoring the models' limitations in handling complex Office document tasks.

Why It Matters

Why should we care about LLMs stumbling over macros or pivot tables? Because the promise of AI is automation, taking over repetitive, time-consuming tasks so humans can focus on more creative and strategic work. Yet, if LLMs can’t reliably automate document tasks, the vision of a smooth AI-driven office remains elusive. Are we expecting too much too soon from these systems?

Looking Forward

There's potential here. The paper's key contribution is highlighting the gap between current capabilities and the lofty goals set for AI in office settings. This isn't just about improving scores. It's about understanding the nuances of human-computer interaction and refining AI to handle real-world complexity. Can future iterations of LLMs close this gap? That's the question AI researchers and developers must now tackle.

Ultimately, while LLMs have made strides in code generation, reliable and nuanced Office automation remains a significant hurdle. As businesses increasingly look to AI solutions, understanding these limitations will be essential for deploying technology that truly enhances productivity.