PaperlayerView exact revision on arXiv

Source-linked AI summary

R$^3$-SQL: Ranking Reward and Resampling for Text-to-SQL

Hojae Han, Yeonseok Jeong, Seung-won Hwang, Zhewei Yao, Yuxiong He

arXiv:2604.25325v1cs.SEcs.AIcs.CL

TL;DR

Text-to-SQL ranking can score equivalent queries inconsistently and cannot recover a correct query missing from its candidate pool. R3-SQL groups candidates by execution result, combines group-level ranking signals, and selectively resamples when coverage is inadequate. It achieves 75.03 EX on BIRD-dev, while its supervised pointwise ranker constrains out-of-domain generalization.

  • Problem

    Functional inconsistency and bounded recall limit ranking-based Text-to-SQL pipelines.

  • Method

    R3-SQL groups candidates by execution result, combines cross-group preference with pointwise group utility, and selectively resamples when the correct SQL is likely absent.

  • Results

    75.03 EX on BIRD-dev is a new state of the art among methods using models with disclosed sizes, with consistent gains across five benchmarks.

  • Takeaways & Limitations

    R3-SQL jointly targets functional consistency and candidate recall in generate-then-rank Text-to-SQL systems.

  • Takeaways & Limitations

    A supervised pointwise ranker constrains out-of-domain generalization, with a 0.46–0.67 performance difference versus the pointwise-ablated variant.

Abstract

from arXiv · show

Modern Text-to-SQL systems generate multiple candidate SQL queries and rank them to judge a final prediction. However, existing methods face two limitations. First, they often score functionally equivalent SQL queries inconsistently despite identical execution results. Second, ranking cannot recover when the correct SQL is absent from the candidate pool. We propose R$^3$-SQL, a Text-to-SQL framework that addresses both issues through unified reward for ranking and resampling. R$^3$-SQL first groups candidates by execution result and ranks groups for consistency. To score each group, it combines a pairwise preference across groups with a pointwise utility from the best group rank and size, capturing relative preference, consistency, and candidate quality. To improve candidate recall, R$^3$-SQL introduces agentic resampling, which judges the generated candidate pool and selectively resamples when the correct SQL is likely absent. R$^3$-SQL achieves 75.03 execution accuracy on BIRD-dev, a new state of the art among methods using models with disclosed sizes, with consistent gains across five benchmarks.

1 Introduction

Modern Text-to-SQL systems generate multiple SQL candidates and rank them, but functional inconsistency and bounded recall limit this paradigm. R3-SQL addresses both through groupwise ranking, agentic resampling, and unified reward modeling, achieving strong benchmark results.

  • Motivation: Existing pointwise and listwise rankers score functionally equivalent SQL queries inconsistently despite identical execution results.This issue is termed functional inconsistency.
  • Motivation: Ranking cannot recover a correct SQL query when the generator omits it from the candidate pool, creating a bounded-recall limitation.Prior resampling solutions from information retrieval remain unexplored in Text-to-SQL.
  • R3-SQL: R3-SQL groups candidates by execution result and ranks groups using pairwise preferences, best representative rank, and group size.The approach combines relative preference, consistency, and candidate quality rather than relying on group size alone.
  • R3-SQL: R3-SQL selectively resamples candidates when an agent predicts that the correct SQL is absent from the pool.This expands candidate recall before ranking.
  • Results: 75.03 execution accuracy on BIRD-dev establishes a new state of the art among methods using models with disclosed sizes.Across five benchmarks, R3-SQL consistently improves execution accuracy over prior ranking-based approaches.
  • Contributions: R3-SQL identifies functional inconsistency and bounded recall as core limitations and reports state-of-the-art performance across five Text-to-SQL benchmarks.The framework also analyzes the contribution of each component.

2 Related Work

Prior Text-to-SQL systems use generate-then-rank pipelines with varied grouping, ranking, and refinement strategies. R3-SQL distinguishes itself by addressing both functional inconsistency and bounded recall, achieving the highest execution accuracy on BIRD-dev among systems with disclosed model sizes.

  • Prior systems: Recent Text-to-SQL systems sample multiple SQL programs and rank or filter them to select a final prediction.Functional inconsistency and bounded recall are identified as key performance challenges.
  • Prior systems: Contextual-SQL uses independent pointwise scoring without grouping, while CHASE-SQL uses pairwise ranking without execution-based grouping.Their respective designs leave functional inconsistency unresolved; CHASE-SQL also does not address semantic bounded recall.
  • Comparison: Table 1 compares systems by execution-result grouping, ranker type, and the condition that triggers resampling or refinement.FMV denotes functional majority voting by group size.
  • R3-SQL distinction: R3-SQL groups candidates by execution result, combines pointwise and listwise signals, and selectively resamples when the pool likely lacks a correct SQL.This contrasts with ranking groups by size alone and with systems that do not address bounded recall.

3 R3-SQL

R3-SQL uses an exploration phase to improve candidate recall and an exploitation phase to rank execution-equivalent groups. Its ranking combines cross-group preferences with pointwise utility while mitigating position bias.

  • 3.1 Improving Functional Inconsistency: R3-SQL first groups SQL candidates with identical execution results, treating each group as one semantic outcome.Aggregating at the group level neutralizes noise from superficial textual differences.
  • 3.1 Improving Functional Inconsistency: The ranking combines cross-group pairwise preference with pointwise utility based on candidate quality and group size.The cross-group signal is primary, while pointwise utility breaks ties when preference margins are unreliable.
  • 3.1 Improving Functional Inconsistency: Pairwise group preference aggregates decisive candidate comparisons across groups and discounts close or ambiguous judgments with a threshold.This produces group ordering that is robust to functional inconsistency.
  • 3.1 Improving Functional Inconsistency: Pointwise group utility uses the strongest representative rank and group size to reflect candidate quality and execution-level self-consistency.Because it is independent of input position, it mitigates positional effects from listwise ranking.
  • 3.2 Mitigating Bounded Recall: An LLM agent audits the initial candidate pool and triggers targeted resampling when it predicts that the correct SQL is absent.The resampled pool is generated from a larger set and reduced to the top-n queries using the pointwise ranker.
  • 3.3 Improving Listwise Ranking: The listwise ranker is trained with a position-consistency objective so correct preferences remain stable under swapped candidate orderings.The reward includes a base correctness term and an auxiliary consistency term.
  • 3.4 Overall Framework: R3-SQL operates as exploration followed by exploitation: resampling expands recall, then group-based ranking resolves functional inconsistency.The overall pipeline executes and groups candidates before combining listwise preferences with pointwise group utility.

4 Experiments

R3-SQL is evaluated across five diverse Text-to-SQL benchmarks using execution accuracy as the main metric and shared rankers across baseline methods. It consistently outperforms the baselines, achieving the highest EX on every dataset and an average EX of 70.65%.

  • Benchmarks: The evaluation covers BIRD, Spider, Spider-DK, EHR-SQL, and ScienceBenchmark, including both cross-domain and domain-specific benchmarks.EHR-SQL and ScienceBenchmark test out-of-domain generalization capabilities.
  • Metrics: Execution accuracy (EX) measures whether predicted and ground-truth SQL produce matching execution results on SQLite.Questions with empty ground-truth execution results caused by database limitations or timeouts are excluded.
  • Baselines: The comparison uses CSC-SQL, Contextual-SQL, CHASE-SQL, and XiYan-SQL as baselines.The same pointwise R3-POINT-32B and listwise R3-7B rankers are used across baseline selection methods.
  • Results: R3-SQL consistently outperforms all baselines, achieving the highest EX on every dataset.It is the only method to break the 70% ceiling, with an average EX of 70.65%.

5 Analysis

The analysis finds that R3-SQL improves consistency, robustness, group scoring, recall, efficiency, and execution accuracy through its ranking and agentic resampling components. Its main supported boundary is reduced pointwise-ranker generalization on out-of-domain benchmarks.

  • 5.1 Functional Inconsistency Mitigation: R3-SQL assigns one score per execution result, reducing score variance among equivalent SQLs to 0.0 by design.Adding the listwise ranker raises EX by +1.56 pp through better alignment between group scores and correctness.
  • 5.2 Reproducibility across Seeds: R3-SQL maintains the lead across four random seeds, with its lowest performance of 73.99 exceeding CHASE-SQL’s peak of 73.27.Baseline rankings fluctuate with candidate-pool variation, whereas R3-SQL remains ahead across seeds.
  • 5.3 Group Scoring Strategy: R3-SQL’s dual-reward group scoring achieves the highest average EX of 70.65.It combines listwise preference and max-based pointwise utility; on in-domain benchmarks it exceeds Listwise by +1.11 pp, while Listwise slightly leads out of domain.
  • 5.3 Group Scoring Strategy: Agentic resampling raises the average ranking upper bound by +3.92 pp, from 78.80 to 82.72.The module expands the candidate pool when the initial pool likely lacks the correct SQL, complementing ranking over a fixed set.
  • 5.4 Position Bias Mitigation: The consistency reward improves input consistency by +11.89 pp over GRPO alone, from 45.60 to 57.49.Removing the pointwise ranker reduces EX by 0.84 pp, from 75.03 to 74.19.
  • 5.5 Computational Overhead: R3-SQL achieves 75.03 EX while reducing inference time by 0.12 sec/query compared with CHASE-SQL.Ranking distinct execution groups and resampling only 37.01% of instances makes it 0.40 sec/query faster than always resampling.
  • 5.6 Ablation Study: Removing agentic resampling lowers BIRD-dev EX from 75.03 to 74.25, while removing execution-group scoring lowers it to 73.47.Using only listwise or only pointwise ranking yields 73.34 and 73.14, respectively.
  • 5.8 Agentic Resampling: Selective resampling achieves 75.03 EX, whereas always resampling improves EX by only +0.07 pp and union-based addition reduces performance to 73.92.The resampling agent reports precision of 93.27 and Skip-class recall of 83.17.

6 Conclusion

R3-SQL addresses functional inconsistency and bounded recall through unified reward modeling, combining groupwise ranking with selective agentic resampling. It achieves 75.03 EX on BIRD-dev, a new state of the art among methods using models with disclosed sizes.

  • R3-SQL addresses functional inconsistency and bounded recall through unified reward modeling.It groups candidates by execution result, combines cross-group preference with single-group utility, and selectively resamples when the correct SQL is likely absent.
  • R3-SQL groups candidates by execution result and ranks groups using cross-group pairwise preference and single-group utility.The utility combines pointwise scoring with group size.
  • 75.03 EX on BIRD-dev establishes a new state of the art among methods using models with disclosed sizes.

7 Limitation

R3-SQL’s supervised pointwise ranker constrains generalization, particularly outside its training domain. The authors report only marginal differences when excluding this module on out-of-domain benchmarks and identify domain-generalized pointwise ranking as future work.

  • A supervised pointwise ranker constrains R3-SQL’s generalization.
  • 0.46–0.67 marginal performance difference appears between R3-SQL and its pointwise-ablated variant on out-of-domain benchmarks.The domain gap in the pointwise ranker is identified as the source of this difference.
  • Integrating a domain-generalized pointwise ranker remains a promising direction for future work.

Appendices

The appendices examine threshold sensitivity, generator robustness, domain generalization, realistic schema settings, efficiency, and ranker selection accuracy. Across these evaluations, R3-SQL maintains strong execution performance while revealing sensitivity to pointwise-ranker domain mismatch.

  • Domain generalization: The pointwise ranker is less reliable on unseen domains because it is fine-tuned exclusively on BIRD.The listwise ranker uses broader training data, while the pointwise module marginally offsets out-of-domain generalization.
  • Ranker validation: 78.86 binary selection accuracy on OmniSQL-7B is +2.78 pp over GRPO.R3-7B consistently outperforms GRPO across different base models.
  • Threshold sensitivity: At low τ, confidence-gated scoring applies the normalized group-mean score only when the pairwise ranker strongly indicates incorrect candidates.Otherwise, the group receives the default score of 1, reducing bias from negative–negative pairings.
  • Generator robustness: 71.83 EX with OmniSQL-7B exceeds Contextual-SQL’s 70.80 by +1.03 pp.The evaluation uses a weaker generator that produces fewer correct candidates.
  • Realistic schema settings: R3-SQL achieves the best EX score of 29.17 on the SQLite subset of Spider2.0.This setting provides complementary evidence under more realistic schema conditions, though it is not a full end-to-end agentic evaluation.
  • Efficiency evaluation: 75.03 EX and 69.73 R-VES are achieved on BIRD-dev, exceeding CHASE-SQL by 1.69 pp in EX and 1.04 pp in R-VES.The results extend beyond execution correctness to the efficiency of selected valid SQL queries.

A.7 Robustness to Grouping Edge Cases

Execution-based grouping has two analyzed edge cases: all-distinct results, which are virtually absent, and all-empty results, which occur occasionally. Agentic resampling detects low-confidence or invalid-content cases and generates new candidates.

  • 0% of BIRD-dev queries fall into the all-distinct execution-result scenario.The authors attribute this to convergence in LLM outputs.
  • 1.43% of queries, or 22/1,534, produce all-empty execution results.This makes grouping uninformative in those cases.
  • Agentic resampling detects low confidence or absent valid content and triggers new candidate generation in all-empty cases.The framework therefore does not fail in these analyzed instances.

B.1 Robustness against Coincidental Correctness

Execution-based grouping can include semantically incorrect SQLs that happen to produce the correct result, creating noise within the target group. R3-SQL addresses this through representative selection and scoring designed to favor semantically valid candidates.

  • False positives are semantically incorrect SQL queries that coincidentally yield the correct execution result and are grouped with correct SQLs.
  • The highest-scoring candidate under the pointwise ranker determines each group’s representative rather than random selection.The pointwise ranker evaluates semantic alignment between the SQL and the question.
  • In two BIRD-dev cases, pointwise ranking selected incorrect candidates because token-level score differences persisted among SQLs sharing execution results.R3-SQL instead groups candidates by execution result and applies listwise ranking over groups, promoting the correct execution group to Top 1.

C Implementation Details

The implementation uses multi-stage candidate generation, ranking, and agentic resampling across several Text-to-SQL benchmarks. It specifies model backbones, sampling settings, prompts, training procedures, evaluation tables, and infrastructure.

  • Candidate Sampling: The initial pool generates 32 SQL candidates per question and selectively resamples 1,024 candidates using a sampling temperature of 0.8.
  • Agentic Resampling: Agentic resampling audits the question, schema, candidate SQL queries, and execution results to determine whether the initial pool contains a correct answer.The auditing agent is trained with generated BIRD-train pools labeled by answer presence, and SFT is adopted for efficiency.
  • Rankers: R3-7B uses OmniSQL-7B as its backbone and trains on positive-negative SQL pairs generated from BIRD-train and Spider-train.Hard negatives are sampled from candidates with high pointwise ranker scores.
  • Rankers: R3-POINT-32B is initialized from Contextual-RM-32B and further optimized on BIRD-train with hard negative SQL candidates.Additional training yields a slight BIRD-dev gain but no consistent trend on other benchmarks.
  • Experimental Setup: Experiments use a single node with 8 NVIDIA H200 GPUs and evaluate splits across five Text-to-SQL benchmarks.The supplied tables compare ranker accuracy, execution accuracy, efficiency, and selection methods across these evaluations.
Loading 2604.25325v1…