Benchmarking AI: FEM-Bench and the Quest for True Physical Reasoning

The quest for artificial intelligence to understand and model the physical world is evolving rapidly, yet a significant challenge remains: rigorous benchmarking of AI's ability to generate scientifically valid physical models. This is where computational mechanics, with its mathematical rigor and numerical precision, plays a essential role.

Why Computational Mechanics Matters

Computational mechanics isn't just a niche academic discipline. It's the backbone of understanding how physical systems behave under forces, deformation, and constraints. The field demands explicit models of physical systems, rigorous reasoning about geometry and spatial relationships, and an understanding of material behavior. In short, it aligns perfectly with AI's budding goals in physical reasoning and world modeling. But do our current AI models meet the mark?

The introduction of FEM-Bench, a computational mechanics benchmark, marks a significant step in evaluating the capability of large language models (LLMs) in this domain. Designed to put AI through its paces, FEM-Bench 2025 presents a series of tasks drawn from a graduate course on computational mechanics. These tasks may seem introductory, but they expose the limitations clearly. What they're not telling you: state-of-the-art models struggle consistently with these challenges.

The State of AI Models

In a series of tests, the Gemini 3 Pro model showed promise, successfully completing 30 out of 33 tasks at least once, while conquering 26 tasks consistently across five attempts. However, creating unit tests, GPT-5 led with an Average Joint Success Rate of 73.8%. That's good, but nowhere near a passing grade if these models plan to stand shoulder to shoulder with human experts.

Other popular models displayed a wide range of performances, underscoring that not all AI is created equal. Let's apply some rigor here: if AI can't consistently tackle the complexities of computational mechanics, how can we trust it in more nuanced applications?

The Road Ahead

FEM-Bench is setting a structured foundation for evaluating AI-generated scientific code, and its future iterations promise to introduce even more sophisticated tasks. This ongoing evolution is essential, as models must continuously adapt and improve to remain relevant.

But here's the million-dollar question: Can AI truly understand the physics underlying the models it generates, or is it merely stringing together code without comprehension? Color me skeptical, but until these models demonstrate solid performance across increasingly complex benchmarks, their utility in real-world applications remains questionable.

The development of benchmarks like FEM-Bench is a critical step in holding AI to a higher standard. It's not enough for AI to perform adequately. it must surpass our expectations and illuminate new possibilities in scientific reasoning and world modeling. Only then can we begin to trust these models to tackle the complexities of the physical world.