SimulCost: Redefining Efficiency in Physics Simulations

Evaluating large language model (LLM) agents in scientific simulations has often overlooked a critical component: the actual costs involved. While token costs have been the main focus, the neglect of tool-use expenses like simulation time and experimental resource consumption has rendered traditional metrics like pass@k impractical under real-world budget constraints.

Introducing SimulCost

Enter SimulCost, the first benchmark designed to emphasize cost-sensitive parameter tuning in physics simulations. This tool pits LLMs against traditional scanning approaches in the area of accuracy and computational costs. Covering an impressive breadth of 2,947 single-round and 1,931 multi-round tasks across 13 simulators, spanning fluid dynamics, solid mechanics, and plasma physics, SimulCost offers a new lens through which to view simulation efficiency.

The specification is as follows: each simulator's cost is analytically defined and platform-independent, making it broadly applicable across different environments. This benchmark isn't just about crunching numbers, it’s about redefining efficiency standards in scientific simulations.

The Reality Check for LLMs

Frontier LLMs, despite their capability, achieve only 46-65% success rates in single-round mode, which plummets to 35-55% when accuracy requirements are high. This statistic alone should prompt a reevaluation of their role in high-stakes environments where precision is non-negotiable. In contrast, multi-round modes see success rates improve to 72-81%, yet they remain 1.5-2.5 times slower than traditional scanning methods. Why invest in models that offer inefficiencies in both time and cost?

This change affects contracts that rely on the previous behavior of LLMs as economical solutions. Until these models can compete with or surpass traditional methods, their adoption in cost-sensitive settings is questionable.

Potential for Knowledge Transfer

SimulCost does more than just highlight existing problems. it explores potential solutions, such as the correlation of parameter groups for knowledge transfer and the influence of in-context examples and reasoning efforts. This might hold the key to making LLMs more practical in the future. The benchmark is open-sourced, providing a static benchmark and an extensible toolkit to aid research focused on cost-aware agentic designs in physics simulations.

Developers should note that without significant advancements, LLMs remain economically unfavorable choices for simulation tasks requiring high accuracy. Is the allure of AI enough to overshadow its current limitations? The future of physics simulations may hinge on answering this question.

Backward compatibility is maintained except where noted below. Researchers and developers are encouraged to dive deeper into the SimulCost toolkit, which is available at https://github.com/Rose-STL-Lab/SimulCost-Bench, and contribute to the evolution of cost-effective simulation environments.