The Limits of AI in Translating Natural Language to Formal Specifications

Transforming natural language into formal specifications is no small feat, especially TLA+, a language critical for verifying complex systems at tech giants like Amazon and Microsoft. Yet, as promising as large language models (LLMs) may seem, their current capabilities fall short in delivering reliable TLA+ specifications without expert intervention.

Understanding the Study

In a recent systematic evaluation, researchers put 30 LLMs to the test, assessing their ability to generate TLA+ specifications from natural language. These models, spanning eight different families, were evaluated using a dataset of 205 TLA+ specifications. The study included both open-weight models using various prompting strategies and proprietary models under few-shot prompting conditions.

The results? A mere 26.6% syntactic correctness paired with a disappointing 8.6% semantic correctness. Even more telling is that these successes were limited to progressive prompting strategies. Surprisingly, model size didn't correlate with performance. For instance, DeepSeek r1:8b outperformed its larger 70B variant, highlighting that bigger isn't always better formal languages.

Where LLMs Fall Short

Despite the size and power of these models, specialized code models underperformed due to negative transfer effects from their training in mainstream languages. This gap signals a need for models that can better align with the reasoning and structure inherent in formal languages like TLA+.

One of the glaring issues is the tendency of LLMs to hallucinate, producing erroneous outputs due to biases in their training data. The study identified five recurring categories of these hallucinations, all of which contribute to the current unreliability of LLM-generated TLA+ specifications.

The Need for Expert Oversight

So, what does this mean for enterprises? In practice, the deployment of AI in generating formal specifications isn't ready for prime time without human oversight. The ROI case requires specifics, not slogans, and the current state of LLMs doesn't deliver on the promise of autonomous specification creation.

Enterprises don't buy AI. They buy outcomes. But when those outcomes require extensive human revision, the total cost of ownership skyrockets. Are we really saving time and resources, or just creating another layer of complexity?

With the evaluation framework, code, and dataset now publicly available, there's hope for improvement and further research. But until then, the consulting deck might say transformation, while the P&L tells a different story.