LLMs and Agents: Architects or Assistants?

The AI-AI Venn diagram is getting thicker. As software systems balloon in complexity, the task of documenting architecture views can't be overlooked. Yet, the manual process is cumbersome, often leaving architects with outdated blueprints. Enter the potential of Large Language Models (LLMs) and agentic approaches to automate this task. Do they deliver?

Testing the Waters

In a strong empirical evaluation, researchers dove into 340 open-source repositories to test the mettle of three LLMs across 13 different experimental setups. The aim was to generate architecture views straight from the source code, using a combination of three prompting techniques and two agentic approaches. The staggering output? 4,137 generated views ready for scrutiny.

But here's where it gets interesting. While prompting strategies slightly improve outcomes, the nuanced art of few-shot prompting reduced clarity failures by 9.2% compared to its zero-shot counterpart. This isn't a partnership announcement. It's a convergence of AI models with promising yet limited results.

Performance Metrics

The custom agentic approach emerged as the frontrunner, demonstrating superior clarity and detail success rates compared to generic agents. It boasted a 22.6% failure rate in clarity and a commendable 50% success in level-of-detail. But despite these numbers, the models consistently operated at the code level, failing to grasp broader architectural abstractions.

So, are these technologies poised to replace human architects? Hardly. The persistent granularity mismatches suggest they're better suited as assistive tools, aiding rather than superseding human expertise. If agents have wallets, who holds the keys to architectural prowess?

The Path Forward

While LLMs and agentic models make strides in architectural view generation, the marriage between human intuition and machine efficiency remains vital. What does this mean for the future of software architecture? Perhaps, a collaborative approach where AI tools augment rather than overtake human architects.

In the grand scheme, these findings underscore a fundamental truth: we're building the financial plumbing for machines, but we're not handing them the blueprints just yet. The future may hold more autonomy for AI-driven architectural design, but for now, the human touch remains indispensable.