Rethinking NL2SQL Metrics: How ROSE Could Revolutionize Accuracy

In the growing domain of Natural Language to SQL (NL2SQL) solutions, the metric Execution Accuracy (EX) has long been the standard. However, this metric is increasingly showing its limitations. Sensitive to syntactical nuances and unable to consider multiple valid interpretations, EX isn't always a reliable measure of a system's effectiveness. It can even be misled by inaccuracies in the so-called 'ground-truth' SQL.

Introducing ROSE: A New Perspective

To counter these limitations, researchers have developed a new metric named ROSE. This intent-centered metric shifts the focus from mere syntactic agreement with ground-truth SQL to evaluating whether the generated SQL actually addresses the underlying question.

ROSE functions through an intriguing Prover-Refuter cascade. The SQL Prover independently checks the semantic accuracy of the predicted SQL against the user's intent. Meanwhile, the Adversarial Refuter serves to challenge this assessment by using the ground-truth SQL as a reference point. It's an approach that seems to marry intuition with rigorous logic.

Why This Matters

The adoption of ROSE could dramatically change how NL2SQL solutions are evaluated. On a specially designed validation set, ROSE-VEC, the new metric has already shown a substantial leap in alignment with human expert judgments, exceeding the next best metric by a striking 24% in Cohen's Kappa.

This isn't just about numbers. It's about making systems that feel more intuitive and aligned with human logic. If the systems we build are judged by their ability to capture user intent rather than their adherence to possibly flawed 'correct' answers, could we see more reliable interaction between humans and machines?

Re-evaluating the NL2SQL Landscape

In an extensive re-evaluation of 19 NL2SQL methods, ROSE revealed four key insights that challenge conventional wisdom. Perhaps the most notable takeaway is the emphasis on user intent over rigid structural correctness. The deeper question here's whether the field will embrace this shift or cling to traditional metrics.

ROSE and its validation set, ROSE-VEC, have been made publicly available, setting the stage for a transformation in NL2SQL research. are significant. If we can accurately assess how well systems understand and fulfill user intent, we might just be on the brink of a new era in AI-driven databases.