PaperlayerView exact revision on arXiv

Source-linked AI summary

ROSE: An Intent-Centered Evaluation Metric for NL2SQL

Wenqi Pei, Shizheng Hou, Boyan Li, Han Chen, Zhichao Shi, Yuyu Luo

arXiv:2604.12988v1cs.DBcs.AI

TL;DR

NL2SQL evaluation is unreliable when reference-dependent metrics penalize valid query variation, overlook ambiguity, or trust flawed ground-truth SQL. ROSE introduces an intent-centered Prover-Refuter cascade and outperforms existing metrics on expert-annotated validation data, while re-evaluating 19 methods to identify broader evaluation patterns.

  • Problem

    Reference-dependent NL2SQL metrics can mishandle ambiguity and erroneous ground-truth SQL, limiting reliable assessment of semantic correctness.

  • Method

    ROSE uses an independent SQL Prover to assess intent alignment and an Adversarial Refuter to challenge that judgment with ground-truth evidence.

  • Results

    ROSE substantially outperforms existing metrics on ROSE-VEC, and the re-evaluation of 19 methods yields four reported insights about models, references, benchmark flaws, and fine-tuning.

  • Takeaways & Limitations

    The authors argue that NL2SQL research should prioritize intent-centered evaluation and improve dataset quality and clarity.

  • Takeaways & Limitations

    ROSE’s reliability depends on the underlying LLM, so metric performance may fluctuate as foundation models change.

Abstract

from arXiv · show

Execution Accuracy (EX), the widely used metric for evaluating the effectiveness of Natural Language to SQL (NL2SQL) solutions, is becoming increasingly unreliable. It is sensitive to syntactic variation, ignores that questions may admit multiple interpretations, and is easily misled by erroneous ground-truth SQL. To address this, we introduce ROSE, an intent-centered metric that focuses on whether the predicted SQL answers the question, rather than consistency with the ground-truth SQL under the reference-dependent paradigm. ROSE employs an adversarial Prover-Refuter cascade: SQL Prover assesses the semantic correctness of a predicted SQL against the user's intent independently, while Adversarial Refuter uses the ground-truth SQL as evidence to challenge and refine this judgment. On our expert-aligned validation set ROSE-VEC, ROSE achieves the best agreement with human experts, outperforming the next-best metric by nearly 24% in Cohen's Kappa. We also conduct a largescale re-evaluation of 19 NL2SQL methods, revealing four valuable insights. We release ROSE and ROSE-VEC to facilitate more reliable NL2SQL research.

1 Introduction

NL2SQL evaluation metrics can misjudge semantic correctness because reference-dependent checks are sensitive to query variation, ambiguity, and flawed ground-truth SQL. ROSE addresses this with an intent-centered Prover-Refuter cascade, validated on ROSE-VEC and applied to 19 methods.

  • Evaluation gap: Execution Accuracy (EX) can produce false negatives up to 28.9% for non-canonical but semantically correct SQL forms.EX also struggles with ambiguous questions and erroneous ground-truth SQL.
  • Evaluation gap: Existing structure-aware and LLM-based metrics reduce some mismatches but remain primarily dependent on a single ground-truth SQL.This limits tolerance for legitimate ambiguity and reference errors.
  • ROSE: ROSE evaluates whether predicted SQL matches question intent through an independent SQL Prover followed by an Adversarial Refuter using ground-truth SQL as evidence.The cascade can identify ambiguous questions and ground-truth errors while reducing reference anchoring.
  • Validation: 24% agreement and 14% accuracy improvements over the closest competitor were reported for ROSE on ROSE-VEC.The result is presented as better alignment with user intent.
  • Re-evaluation: A re-evaluation of 19 NL2SQL methods identifies four patterns involving base-model capability, semantic-reference divergence, benchmark flaws, and fine-tuning.The reported benchmark flaws include ground-truth errors and question ambiguities.
  • Validation: ROSE-VEC contains 585 expert-consensus samples for rigorous validation of NL2SQL metrics.The dataset is released with detailed annotations.

2 Preliminary

NL2SQL generates predicted SQL from a question and database, while ideal evaluation requires both syntactic validity and semantic alignment with user intent. Perfect semantic assessment is computationally infeasible, motivating practical reference-based approximations.

  • Task formulation: NL2SQL maps a natural-language question and database contents to predicted SQL and compares its execution results with ground-truth results.The notation distinguishes predicted and ground-truth SQL and their respective result sets.
  • Ideal evaluation: Executable SQL can still be incorrect when it violates acceptance criteria for schema validity or alignment with the question.Criteria may include user-specific tolerance for duplicates or NULL values.
  • Ideal evaluation: The ideal judgment function decomposes intent evaluation into syntactic validity and semantic correctness.The two conditions are represented by σsyn and σsem.
  • Evaluation limits: Perfect semantic correctness assessment is computationally infeasible, so practical metrics approximate it through SQL structure or execution-result similarity.These approximations compare the prediction with the ground-truth SQL.

3 Related Work

NL2SQL metrics include deterministic rule-based measures and LLM-based semantic judges. Deterministic methods compare strings, components, structures, or results, while LLM-based methods use reasoning but generally remain ground-truth dependent.

  • Metric categories: The evaluation landscape is divided into deterministic and LLM-based metrics.Formal definitions are provided in the paper’s appendix.
  • Deterministic metrics: Deterministic metrics use predefined rule-based algorithms to evaluate query correctness.They form the foundation of standard NL2SQL benchmarks.
  • Deterministic metrics: Exact Match requires normalized predicted SQL to be character-for-character identical to ground-truth SQL.Component Match instead provides partial credit at the clause level.
  • Deterministic metrics: Execution Accuracy compares execution results, while Enhanced Tree Match compares normalized abstract syntax trees.Other deterministic measures compare keyword sets or probe specific SQL capabilities.
  • LLM-based metrics: LLM-based metrics aim to use model reasoning for more realistic utility judgments but still commonly rely on ground-truth SQL.Examples include LLM-SQL-Solver and FLEX.

4 Methodology

ROSE uses a staged scoring cascade that first requires executable SQL, then applies independent semantic assessment and adversarial challenge when execution results differ. The Refuter uses ground-truth SQL as counter-evidence rather than as a matching reference.

  • SQL Prover: SQL Prover evaluates a syntactically valid prediction independently against the question’s intent and acceptance criteria when execution results differ.It outputs a boolean judgment and rationale.
  • Adversarial Refuter: Adversarial Refuter challenges SQL Prover’s affirmative judgment using ground-truth SQL as evidence rather than as a reference to match.This addresses the risk that an independent Prover becomes overly permissive.
  • Cascade cases: When execution results match, the Refuter compares predicted and ground-truth reasoning to detect coincidental correctness and erroneous ground truth.It acts as a safeguard against false positives.
  • Cascade cases: When execution results differ, Refuter arbitration may overturn Prover approval, flag ground-truth error, or accept both queries as ambiguous interpretations.The decision is reevaluated against the user’s intent.
  • ROSE scoring: A prediction must be executable, pass SQL Prover when results differ, and withstand Refuter challenge to earn ROSE score 1.Failure at any stage produces score 0.

5 Experiments: Validating ROSE

ROSE is evaluated against expert judgments on ROSE-VEC using multiple validation metrics and model backbones. It consistently outperforms existing metrics and also supports diagnostic labeling of erroneous ground-truth SQL and ambiguous questions.

  • Validation setup: ROSE-VEC contains 585 human-labeled NL-SQL pairs from Spider Test and BIRD Dev outputs.Each output is independently judged by two of five experts.
  • Validation setup: Cohen’s Kappa is the primary validation metric, supplemented by accuracy, Matthews correlation, and F1.The metrics compare NL2SQL judgments with expert labels, treating “correct” as the positive class.
  • Compared metrics: The evaluation compares deterministic metrics, LLM-based metrics, and ROSE w/o Refuter across three reasoning-model backbones.The LLM-based backbones are OpenAI o3-2504, Gemini-2.5 Pro-2506, and DeepSeek-R1-2505.
  • Metric effectiveness: 25.56% Cohen’s κ for EX confirms a large gap between deterministic evaluation and expert judgment.Existing LLM-based metrics improve on deterministic metrics but remain below expert-level agreement, while ROSE w/o Refuter surpasses them.
  • Metric effectiveness: 80.43% Cohen’s κ for ROSEo3-2504 is 24% higher than FLEXo3-2504, with ROSE achieving the best performance across validation metrics and three backbones.The gain from ROSE w/o Refuter to the full cascade indicates that adversarial use of ground-truth SQL contributes to peak performance.
  • Diagnostic capability: 84.32% precision for GoldX and 91.23% for AmbQ show that ROSE can reliably flag ground-truth SQL errors and ambiguous questions.These diagnostic labels support automated dataset analysis and cleaning, extending ROSE beyond scoring.

6 Experiments: Benchmarking

The re-evaluation compares EX and ROSE across NL2SQL methods, showing that base-model capability dominates performance while benchmark flaws widen metric disagreement and fine-tuning narrows it through stylistic alignment.

  • Base Model Dominance: Base-model capability, rather than system-level engineering, is the primary driver of NL2SQL performance.Systems cluster into performance tiers defined by their base model, with performance scaling across model generations.
  • Base Model Dominance: Performance scales monotonically from GPT-3.5 through GPT-4, GPT-4o, and GPT-5, and engineered systems inherit this ordering.For example, RSL-SQL based on GPT-4o outperforms C3-SQL based on GPT-3.5 in both metrics.
  • Widening Gap: As models become more powerful, EX increasingly diverges from ROSE as an indicator of semantic correctness.For prompting methods, the gap grows from less than 5% in mid-2023 systems to more than 20% in models projected for mid-2025.
  • Benchmark Flaws: Ground-truth errors and ambiguous questions are major sources of disagreement between EX and ROSE.Discordance exceeds 80% for erroneous ground-truth SQL and remains around 60% for ambiguous questions, compared with less than 20% overall.
  • Benchmark Flaws: GoldX and AmbQ account for roughly 45% and 10% of discordant cases, respectively, together explaining more than half of EX–ROSE divergences.These findings identify ground-truth correctness and question clarity as important dataset concerns.
  • Fine-Tuning Effect: Fine-tuning narrows the ROSE–EX gap by encouraging models to mimic dataset-specific SQL style.Prompting methods show a larger gap across difficulty levels, with roughly four times the gap of fine-tuned methods on Simple and Moderate questions.

7 Conclusion

The paper introduces ROSE, an intent-centered metric using an adversarial Prover-Refuter cascade, and evaluates it against expert judgments. Its re-evaluation identifies base-model dominance, metric divergence from benchmark flaws, and fine-tuning’s stylistic alignment effect.

  • Conclusion: ROSE is an intent-centered metric designed to align NL2SQL evaluation with expert judgment.It uses an adversarial Prover-Refuter cascade and is evaluated on ROSE-VEC, a publicly released dataset of 585 expert-annotated samples.
  • Conclusion: The re-evaluation finds that base models dominate system engineering as drivers of performance, while EX increasingly diverges from semantic correctness.It attributes the gap primarily to erroneous ground-truth SQL and question ambiguity.
  • Conclusion: Fine-tuning may teach stylistic conformity to the current training set rather than improve semantic reasoning.The conclusion frames this as a suggested interpretation of the re-evaluation findings.
  • Conclusion: The findings support developing intent-centered metrics and improving dataset quality and clarity for more reliable NL2SQL evaluation.The paper presents ROSE as a possible path beyond matching a single potentially flawed ground-truth solution.

Limitations

ROSE’s reliability depends on its underlying reasoning models, its consensus-filtered validation set, and substantially greater computational resources than deterministic metrics. Version management mitigates model drift, but these constraints remain important boundaries.

  • Model dependence: ROSE’s efficacy varies with the reasoning capabilities of its underlying foundational LLMs.Different backbone models produce different performance, so reliability may fluctuate as new models are released.
  • Model dependence: Backbone updates can improve expert alignment but also introduce score drift across model versions.The paper versions judge configurations and requires candidate models to pass re-validation before adoption.
  • Validation-set bias: ROSE-VEC may overrepresent clear cases because it retains only instances with exact agreement between two annotators.Borderline, disagreement-prone, and genuinely ambiguous queries may be under-represented.
  • Validation-set bias: Reported agreement on ROSE-VEC may not fully reflect performance across the broader distribution of NL2SQL outputs.The limitation follows directly from the consensus-filtered construction of the validation set.
  • Cost and latency: ROSE is substantially more resource-intensive and slower than deterministic metrics such as Execution Accuracy.The overhead may limit rapid, iterative method development, although call routing, concise prompts, and parallelism reduce it.

F Additional Ablation Studies

Ablations show that ROSE’s full Prover-Refuter cascade outperforms variants that remove ground-truth supervision or collapse its stages. ROSE also transfers across open-source backbones, while parallel execution substantially improves evaluation speed.

  • Ablation studies: Removing ground-truth supervision or collapsing verification and critique into one prompt reduces agreement with expert judges.Across both OpenAI o3-2504 and Gemini-2.5 Pro-2506, every ablated variant trails the full cascade.
  • Ablation studies: The full ROSE cascade requires both reference-SQL access and multistage Prover-Refuter reasoning for its strongest validation performance.The ablations identify both design choices as important components.
  • Open-source backbones: ROSE consistently improves over FLEX across all four validation metrics with three open-source backbones.The evaluated backbones include Qwen3-235B-A22B-Thinking-2507, Qwen3-30B-A3B-Thinking-2507, and DeepSeek-R1-Distill-Qwen-32B-2501.
  • Time efficiency: Parallel execution substantially reduces ROSE’s end-to-end evaluation time.This result is reported for ROSEo3-2504 on ROSE-VEC-BIRD under different thread counts.
  • Time efficiency: ROSE has higher single-thread per-question latency than FLEX, but multithreading lowers effective latency for parallel benchmarking.The paper therefore presents parallel execution as important for practical use in benchmarking settings.

H.2 Cost Efficiency

ROSE remains cost-efficient in practice despite its possible second-stage Refuter calls, because conditional invocation and concise prompts limit average inference usage.

  • Monetary cost: ROSE remains cost-efficient in practice despite potentially invoking a second-stage Refuter.Concise prompts and conditional second-stage calls help control monetary cost relative to FLEX.
  • Call usage: More than half of questions complete with one LLM call, while ROSE averages 1.45 calls per question.These call patterns explain how the cascade controls cost despite having a Refuter stage.
  • Failure audit: ROSE’s disagreement audit identifies 29 failure cases, including 26 false negatives and 3 false positives.The distribution suggests ROSE is generally conservative and more likely to under-credit than over-credit.

I.1 Logical Robustness vs. Coincidental Correctness

ROSE distinguishes logically robust SQL from queries that match the reference only under the current database state, while also exposing disagreements caused by representation strictness and ambiguity.

  • Logical robustness: 15/29 disagreements involve SQL that coincidentally returns the correct current result but is logically fragile under plausible data variations.ROSE penalizes such queries because execution coincidence does not establish logical robustness.
  • Logical robustness: A query counting loan rows can match the gold answer while over-counting accounts once accounts hold multiple contracts.ROSE flags this logically unsafe construction despite its current execution match.
  • Units and format: 11/29 disagreements arise from strictness about representation, including units, scaling, and output conventions.Annotators may accept equivalent representations that ROSE treats as semantically incomplete.
  • Units and format: A stored proportion in [0, 1] is marked incorrect when the question requests a percentage.Annotators accept the proportional form up to deterministic scaling, whereas ROSE treats the unit mismatch as a semantic gap.
  • Ambiguity: 3/29 disagreements involve domain interpretation, schema heuristics, or genuine ambiguity where multiple readings can be defensible.ROSE may enforce a more literal interpretation than annotators in these cases.

J.1 Metric effectiveness

ROSE provides stronger expert-aligned evaluation than deterministic metrics across ROSE-VEC-BIRD and ROSE-VEC-Spider. Its refuter stage improves agreement-oriented performance, while diagnostic precision varies by backbone and split.

  • Metric effectiveness: ROSE achieves the best agreement and correlation within each backbone on both ROSE-VEC-BIRD and ROSE-VEC-Spider.Deterministic metrics lag behind LLM-based judges, although EX consistently exceeds EM and ETM.
  • Metric effectiveness: ROSE improves κ and MCC over ROSE w/o Refuter on both subsets, especially on ROSE-VEC-Spider.The refuter contribution is strongest where deterministic signals are weakest.
  • Metric effectiveness: ROSE w/o Refuter attains slightly higher F1 than ROSE only for Gemini-2.5 Pro on ROSE-VEC-BIRD, while ROSE retains superior agreement-oriented scores.This is the reported exception to ROSE's broader refuter-stage advantage.
  • Diagnostic labelling: 97.56% AmbQ precision is achieved by OpenAI o3 on ROSE-VEC-Spider.On both splits, o3 leads GoldX and AmbQ precision; AmbQ detection is substantially harder on BIRD for Gemini-2.5 Pro and DeepSeek-R1.

K.2 Fine-tuned Methods

Fine-tuned NL2SQL methods improve task proficiency through training on large text-to-SQL corpora, while evaluated systems use varied generation, search, revision, and verification strategies. The evaluation materials emphasize intent-sensitive judging, ambiguity tolerance, and direct comparison of predicted and gold SQL execution outputs.

  • Fine-tuned Methods: Fine-tuned methods train language models on large-scale text-to-SQL corpora to improve proficiency and robustness.Examples include OmniSQL-32B, CodeS-15B, CHESS, and RESDSQL-3B.
  • Fine-tuned Methods: OmniSQL-32B fine-tunes Qwen2.5-Coder on SynSQL-2.5M and incorporates Spider and BIRD training sets for broader SQL coverage.The stated goal is improved robustness across SQL dialects and complexities.
  • Fine-tuned Methods: CodeS-15B combines SQL-centric pre-training with instruction tuning to improve schema linking and complex join generation.It builds on StarCoder and uses diverse text-to-SQL corpora.
  • Evaluation comparison: ROSE−EX gaps are positive for several strong systems but small or negative for some fine-tuned models, showing that execution signals can under- or over-estimate semantic correctness.GPT-4 and GPT-4o show sizable positive gaps, whereas CodeS and RESDSQL show small or negative gaps.
  • Intent-sensitive evaluation: The SQL Prover judges logical correctness against question intent, while execution results are auxiliary rather than decisive.Its reasoning order proceeds from expected answer content to the predicted SQL's purpose and then to whether results satisfy the question.
  • Ambiguity handling: ROSE accepts clearly committed interpretations when question wording is ambiguous and schema or evidence does not contradict them.The judging principles also permit multiple reasonable relation mappings and avoid penalizing NULLs or duplicates unless the question requires otherwise.
Loading 2604.12988v1…