PaperlayerView exact revision on arXiv

Source-linked AI summary

ReToolSQL: Agentic Reinforcement Learning for Robust Text-to-SQL

Pratik Kakkar, Chandra Dhir, Ravi Shankar, Pareekshit Reddy Gaddam, Anup Shirgaonkar

arXiv:2608.27796v1cs.AI

TL;DR

Text-to-SQL systems often generate one query and cannot iteratively recover from execution errors, motivating a single dense model that can verify and repair SQL. ReToolSQL combines supervised warm-starting with agentic reinforcement fine-tuning over multi-turn database-tool trajectories, reaching 74.32% single-pass EX and 74.77% EX with self-consistency on BIRD-SQL development.

  • Problem

    Existing text-to-SQL methods often treat generation as single-turn, limiting explicit iterative use of execution feedback for error recovery.

  • Method

    ReToolSQL uses verified rejection-sampled traces for SFT, followed by agentic RL over multi-turn trajectories with database tools and composite rewards.

  • Results

    74.32% single-pass EX and 74.77% EX with self-consistency are achieved on the BIRD-SQL development benchmark by SFT→RFT.

  • Takeaways & Limitations

    SFT expands pass@k coverage of hard questions, while agentic RL sharpens single-pass accuracy within a single dense 31B model.

  • Takeaways & Limitations

    The formulation assumes access to database tools and enriched schema representations during the tool-augmented decision process.

Abstract

from arXiv · show

Recent work has shown that reinforcement learning from execution feedback can substantially improve text-to-SQL performance, often enabling smaller models to match or exceed much larger systems. However, most existing approaches treat SQL generation as a single-turn task, limiting the model's ability to recover from errors through iterative refinement. We present ReToolSQL, a two-stage training framework for text-to-SQL that combines (i) a supervised warm-start on rejection-sampled reasoning traces with (ii) agentic reinforcement fine-tuning (RFT) over multi-turn tool-use trajectories. The key insight is that the two stages act on complementary axes, the supervised fine-tuning (SFT) on verified privileged-teacher traces expands the set of solvable questions (raising pass@k coverage on the hardest cases), while RFT converts that expanded capability into higher single-pass accuracy by teaching the model when to verify, what evidence to retrieve, and how to repair faulty SQL from execution feedback. Applied to Gemma 4 instruction-tuned (31B), RFT alone achieves 73.66% execution accuracy (EX) on the BIRD-SQL development benchmark (74.12% EX with self-consistency). Initializing RFT from the SFT checkpoint (SFT$\to$RFT) yields our strongest model at 74.32% EX single-pass and 74.77% EX with self-consistency. At the time of writing, this ranked first on the BIRD single-model development-set leaderboard. The approach uses composite rewards anchored on execution correctness, requires no human annotation beyond the benchmark itself, and operates within a single dense 31B model, showing that a properly designed SFT$\to$RFT pipeline over tool-use trajectories is a practical path toward robust enterprise-grade text-to-SQL.

1 Introduction

ReToolSQL addresses brittle single-pass text-to-SQL by training a dense model to use database tools across multi-turn propose–verify–repair trajectories. Its two-stage SFT→RFT approach combines verified warm-start traces with agentic reinforcement learning, achieving the paper’s strongest reported BIRD-SQL development results.

  • Motivation: Single-pass text-to-SQL remains brittle because one incorrect join, literal, or aggregation can cause execution failure without a recovery mechanism.Realistic deployment also involves large schemas, noisy values, and numerical or temporal reasoning.
  • Motivation: Existing methods commonly treat SQL generation as a single-turn act, leaving execution feedback underused for iterative reasoning.The paper frames this limitation as a question of whether one dense model can propose, verify, and repair SQL through tool interaction.
  • Approach: ReToolSQL trains agentic reinforcement learning over multi-turn tool-use trajectories with a sandboxed SQL executor, column profiler, and BM25 value search.The method adds schema-linking rewards so tool use is grounded in database-specific evidence.
  • Results: 73.66% execution accuracy is achieved by agentic RL alone on BIRD-SQL development, rising to 74.12% with self-consistency.These results use a Gemma 4 instruction-tuned 31B model.
  • Results: 74.32% single-pass and 74.77% with self-consistency are achieved by the SFT→RFT model, improving over both individual stages.The supervised stage expands pass@16 coverage on hard questions, while RL converts that coverage into higher single-pass accuracy.
  • Implications: A single dense 31B model ranks first on the BIRD-SQL single-model development leaderboard.The paper presents the pipeline as a practical route toward robust enterprise-grade text-to-SQL without a multi-model system.

2 Related Work

ReToolSQL connects text-to-SQL with execution-grounded RL, agentic tool use, and test-time scaling while targeting the complexity of heavyweight orchestration pipelines. Its distinction is to learn tool proficiency directly from complete trajectories within one dense model.

  • Text-to-SQL Benchmarks: BIRD evaluates text-to-SQL on 12,751 question–SQL pairs across 95 noisy databases and more than 37 domains using execution accuracy.This evaluation emphasizes predicted-versus-gold query outputs rather than string or tree similarity.
  • Pipelines and Scaling: Recent BIRD leaders use candidate generation, selection, specialized agents, or iterative orchestration to increase test-time compute and accuracy.The cited systems range from four-agent pipelines to multi-generator ensembles and tournament-style selection.
  • RLVR: RLVR methods use verifiable rewards, while GRPO estimates advantages from reward statistics over sampled completion groups without a value network.DAPO adds dynamic sampling to discard homogeneous prompt groups and other training stabilizers.
  • ReToolSQL: ReToolSQL applies DAPO-style dynamic sampling, composite schema-aware rewards, and multi-turn trajectory rewards to text-to-SQL.The composite reward retains schema-linking signal when execution rewards are uniform within a group.
  • ReToolSQL: Unlike pipeline systems, ReToolSQL makes tool proficiency emerge from gradient updates on complete trajectories rather than from prompts or external orchestration.The system combines agentic tool use, GRPO with dynamic sampling, schema-aware rewards, and execution-based self-consistency in one dense 31B model.

3 Methodology

ReToolSQL formulates text-to-SQL as a tool-augmented sequential decision problem in which a single language model uses schema evidence and read-only database tools before producing SQL. GRPO with DAPO-style optimization trains multi-turn trajectories using execution-grounded rewards, masked tool observations, and verification-oriented protocols.

  • Problem formulation: A single dense 31B model receives a question, schema, optional hint, and fixed read-only tools, then emits tool calls or a terminal SQL answer.Episodes terminate at <final_answer> or after a maximum of 8 tool turns.
  • Schema-grounded observation space: M-Schema enriches each column with type, meaning, key and nullability flags, statistics, representative values, and foreign-key paths for evidence-based schema linking.The encoding is designed to expose data evidence in a compact, regular layout.
  • Turn protocol: Every assistant turn uses a bounded scratch pad followed by exactly one tool call or final answer, with explicit table and column annotations before finalization.The annotations expose schema-linking decisions to the linking rewards.
  • Read-only tools: The workflow drafts SQL, executes it with sqlite_query, and invokes sqlite_peek or bm25_search_sqlite only to repair detected faults.The tools provide execution errors, column profiles, and value searches for verification and grounding.
  • Policy optimization: GRPO samples G = 16 trajectories per question and computes group-relative advantages from composite rewards, avoiding a learned value function.The DAPO objective uses asymmetric clipping and a scheduled KL penalty toward a frozen reference policy.

3.5 Dynamic Sampling

The training and inference design addresses sparse or degenerate execution feedback by retaining heterogeneous rollout groups, masking environment observations during optimization, shaping length and auxiliary rewards, and applying execution consensus at test time. The SFT warm-start expands hard-question coverage, while RFT converts that coverage into stronger single-pass performance.

  • Dynamic sampling: Candidate rewards are computed before trainer-side log-probability evaluation, directing expensive computation toward informative groups.This makes oversample-and-replace selective rather than processing every candidate through the full trainer pass.
  • Tool-observation loss masking: Tool-observation masking excludes environment-injected rows and profiles from the ratio and loss while keeping them visible as attention context for later turns.Gradients therefore target policy reasoning, tool calls, and final SQL rather than memorization of database contents.
  • Length regularization: The continuous length penalty is 0 through Ltarget = 6,000 tokens, decreases linearly to −α by Lmax = 8,000, and remains −α beyond that limit.Here α = 0.5 reward units.
  • Composite rewards: The composite reward is execution-dominated, with auxiliary linking terms preserving non-flat advantages when execution rewards collapse; its weights are heuristic and untuned.Result equivalence uses unordered raw-row set comparison with a 30-second per-query timeout.
  • Inference-time scaling: Execution consensus samples N = 16 trajectories, discards errors and empty results, and returns the largest execution-result cluster with deterministic tie-breaking.The rule concentrates probability on agreeing executions because incorrect programs tend to scatter across clusters.
  • SFT warm-start: SFT→RFT improves over applying reinforcement fine-tuning directly because verified warm-start traces expand coverage where base-policy rollout groups are otherwise all-wrong.Hard examples use privileged-teacher traces, while easier examples use the model’s own correct trajectories; the resulting checkpoint initializes GRPO.

4 Experimental Setup

ReToolSQL is evaluated on the challenging BIRD-SQL development benchmark using Gemma 4 31B Instruct and controlled configurations of tools, training, and inference. Accuracy is measured by execution-based set equality under a fixed timeout and parallel evaluation protocol.

  • Benchmark: BIRD-SQL spans 11 large databases with complex schemas, noisy values, and domain-specific terminology.The full development set contains 1,534 questions across 11 databases, while RL training uses 6,601 questions from 69 disjoint databases.
  • Evaluation: Execution Accuracy (EX) counts a prediction as correct only when predicted and gold SQL produce identical unordered tuple sets.Evaluation uses a 30-second per-query timeout and eight parallel workers, with separate reporting for Simple, Moderate, and Challenging questions.
  • Models and baselines: The primary model is Gemma 4 31B Instruct, a dense decoder-only transformer, with Gemma 4 E4B Instruct used for scale ablations.The experiments also isolate tool access, RL training, and inference-time scaling under controlled configurations.
  • Infrastructure: Training runs on one node with eight NVIDIA H200 GPUs, splitting policy optimization across six GPUs and rollout generation across two.The rollout server uses vLLM with tensor parallelism of two and a 24,576-token context window.
  • Training configuration: The RL configuration uses a 10−6 AdamW learning rate, effective batch size 192, temperature 1.2, nucleus sampling p = 0.95, and one training epoch.Additional settings include a 0.03 warmup ratio and gradient clipping at 1.0.

5 Results

The results isolate supervised warm-starting, agentic reinforcement fine-tuning, and their combination on BIRD-SQL. SFT broadens hard-question coverage, while RFT improves single-pass reliability, making SFT→RFT the strongest configuration.

  • Main results: 74.32% EX single-pass and 74.77% EX with self-consistency are achieved by the compound SFT→RFT model on BIRD-SQL development.The evaluation covers 1,534 questions under a common protocol with tool access.
  • SFT alone: 72.69% temp-0 accuracy and 81.29% pass@16 coverage are achieved by SFT alone, compared with 76.9% base pass@16 coverage.The warm-start uses execution-verified rejection-sampled traces to add correct solutions for hard questions.
  • Agentic RFT alone: 73.66% execution accuracy is achieved by agentic RFT alone, rising to 74.12% with self-consistency.RFT improves temp-0 accuracy by 2.47% versus SFT’s 1.50% while leaving pass@16 coverage essentially unchanged at 77.2%.
  • SFT→RFT: 74.32% temp-0 EX and 74.77% self-consistency EX are achieved by SFT→RFT, with pass@16 held at 81.94%.Relative to RFT from the base model, this is +0.66% temp-0 and +0.65% self-consistency at parity-or-better pass@16.
  • Leaderboard comparison: 74.77% with self-consistency places the compound model ahead of Gemini-SQL2 at 74.12% on the BIRD single-model development leaderboard.The model reaches the top position using a single dense 31B model without an external retriever, learned selector, or multi-component pipeline.

6 Analysis

Analyses examine how tools, RL, sampling, model scale, regularization, and question difficulty shape performance. The findings emphasize interaction between tool access and RL, limited sampling diversity at 31B, and stronger RL gains for smaller models.

  • Component isolation: 3.97% is the gain over base when RL training and tool access are combined, exceeding the sum of individual contributions by 1.50%.Tool access alone improves the base model by 2.02%, while RL without tools yields 0.45%.
  • Pass@k and sampling: 73.66% pass@1, 74.12% self-consistency, and approximately 77.2% pass@16 are reported for ReToolSQL.The 3.08% gap between self-consistency and the pass@16 upper bound indicates unrealized candidate-selection potential.
  • Pass@k and sampling: 3.37 unique SQL strings are produced per 16 candidates on average by the 31B policy, limiting self-consistency gains to +0.46%.Candidates tend to cluster around one correct or incorrect formulation.
  • Scale comparison: 68.45% is achieved by E4B after RL versus 65.12% before RL, while 31B rises from 71.71% to 73.66%.The smaller model gains 3.33%, compared with 1.95% for 31B, under greedy decoding with tools.
  • Training dynamics: 2.09% better greedy accuracy and 1.96% better self-consistency accuracy result from the decaying beta schedule.The schedule preserves entropy approximately twice as long, declining from 0.40 to 0.20 instead of 0.40 to 0.10 without KL regularization.
  • Difficulty analysis: RL progressively improves Moderate and Challenging tiers, including a 5.19% gain on Moderate questions.After agentic RL with tools, Challenging accuracy reaches 75.32%, producing more balanced tier performance.

7 Conclusion

ReToolSQL trains a single dense 31B model with tool-augmented reinforcement learning, combining execution-based verification with a supervised warm-start. The resulting SFT→RFT pipeline improves both difficult-question coverage and single-pass accuracy while retaining a simple serving architecture.

  • The system trains one dense 31B model with a sandboxed SQL executor, column profiler, and BM25-based value searcher.The policy learns a propose–verify–repair loop grounded in database evidence.
  • 74.32% single-pass EX and 74.77% EX with self-consistency are achieved by the strongest SFT→RFT configuration on BIRD-SQL development.The configuration improves over both individual training stages.
  • SFT expands pass@k coverage on hard questions, while agentic RL sharpens single-pass accuracy.The two stages therefore act on complementary axes.
  • Tool access alone provides +2% gains, whereas RL over tool-use trajectories provides +4%.
  • A single-policy architecture competes with multi-component pipelines without an external retriever, learned selector, or mixture-of-experts routing.Inference requires only a single GPU for the 31B model under standard serving frameworks.

Appendix A Worked Example: Multi-Turn Tool Trajectory

The worked trajectory illustrates ReToolSQL’s multi-turn propose–verify–repair behavior on california_schools. Execution feedback triggers error detection, targeted evidence retrieval, denominator profiling, and guarded query repair.

  • The california_schools trajectory demonstrates execution-triggered error detection, targeted evidence retrieval, and structured repair with safety guards.
  • The policy drafts SQL and executes it before continuing to the next turn.Each tool call ends the turn and is followed by an environment observation.
  • BM25 search retrieves the correct literal after an empty result is detected.
  • Column profiling reveals nulls in the denominator column before the policy emits a guarded final query.

Appendix B Execution-Based Self-Consistency

Execution-based self-consistency samples multiple SQL trajectories, filters invalid outputs, and selects the largest execution-result cluster. This favors semantically stable answers over isolated generations.

  • 16 candidate trajectories are sampled at temperature 1.2 and validated by execution under training-time safety constraints.
  • Candidates that error or return empty results are discarded before result-set clustering.
  • The representative SQL from the largest result-set cluster is returned, with ties broken by sample order and then shorter SQL length.
  • Consensus improves robustness by favoring semantically stable answers over isolated generations.Distinct hallucinations tend to scatter across singleton clusters, while correct formulations converge on shared outputs.

Appendix C Tool Definitions

The tool API defines concrete interfaces, resource limits, and returned structures for ReToolSQL’s database interactions. All three tools are read-only during training and inference.

  • Table 7 specifies call signatures, parameter names, defaults, sandbox resource limits, and returned structures for each tool.
  • All three database tools enforce read-only access, preventing schema modifications and data writes.This restriction applies during both training and inference.

Appendix D Entropy Dynamics without KL Regularization (Gemma E4B model)

Without KL regularization, policy entropy collapses early and accuracy soon plateaus, while a decaying KL schedule preserves exploration and supports continued improvement.

  • No-KL dynamics: 0.40 to 0.10: policy entropy falls within the first 40 steps without a KL penalty and remains low thereafter.The no-KL run uses β = 0.
  • No-KL dynamics: Greedy and self-consistency accuracy plateau after entropy collapse, while pass@k headroom is largest at the earliest checkpoints.Pass@k headroom shrinks after accuracy plateaus.
  • Decaying KL schedule: β = 0.005 → 0.001 → 0: a decaying KL schedule slows entropy decline and prevents premature collapse.The schedule applies β = 0.005 for steps 0–40, β = 0.001 for steps 40–80, and β = 0 thereafter.
  • Decaying KL schedule: Execution accuracy and self-consistency continue improving beyond the checkpoint where the no-KL run plateaus.The early KL penalty keeps the policy exploratory during training.
Loading 2608.27796v1…