Do Large Language Models Really Understand Rules?

Large Language Models (LLMs) are often hailed for their ability to mimic human reasoning. But a closer look at their capabilities raises a fundamental question: do they genuinely understand explicit rules or merely rely on statistical regularities from their vast pretraining datasets?

Introducing PLSemanticsBench

The study in question dives into this issue by using program execution as a test case, where the behavior is defined through symbolic transition rules. The researchers have introduced PLSemanticsBench, a novel benchmark designed to probe whether LLMs can condition their reasoning on formal semantics.

PLSemanticsBench pairs simple C programs with two distinct semantic systems. Specifically, small-step operational semantics and K semantics are employed. The study examines four specific capabilities of the models: composing rules to reach final states, choosing appropriate rules when states remain unchanged, maintaining conditioning over extended sequences, and adhering to new semantics with provided rules.

Challenging the Models

To truly test the limits of LLMs, the researchers didn't just stick to familiar grounds. Instead, they redefined known operators to create symbol-meaning conflicts and introduced novel symbols explained only via the provided rules. The models were stress-tested across Human-Written, LLM-Translated, and Fuzzer-Generated scenarios, each presenting increasing structural complexity.

The results? Notably concerning. While some models reached up to 90% accuracy under standard semantics, their performance plummeted by 40-60 percentage points when faced with semantic mutations or increased complexity. This stark drop suggests a reliance on pretrained associations rather than an understanding of formal rules.

A Reality Check

What does this mean for the future of LLMs? The data shows that only a few models could maintain non-zero long-horizon conditioning accuracy, with the best reaching merely 35%. This is a clear indicator that current LLMs often default to pattern recognition over systematic rule conditioning.

Should we be surprised? Perhaps not. The benchmark results speak for themselves. However, it's a wake-up call for those who might overestimate the current state of AI. How can we expect LLMs to revolutionize fields like law or medicine if they can't reliably follow explicit rules?

The paper, published in Japanese, reveals that there's still a long way to go before LLMs can truly understand and apply complex formal semantics as a human would. Western coverage has largely overlooked this critical gap, focusing instead on surface-level achievements.

, while LLMs show promise, they still fall short in areas requiring genuine understanding of formal semantics. As AI continues to evolve, the challenge will be in bridging this gap and ensuring these models can go beyond pattern recognition to true rule-based reasoning.