Source-linked AI summary
Reasoning-SQL: Reinforcement Learning with SQL Tailored Partial Rewards for Reasoning-Enhanced Text-to-SQL
Mohammadreza Pourreza, Shayan Talaei, Ruoxi Sun, Xingchen Wan, Hailong Li, Azalia Mirhoseini, Amin Saberi, Sercan "O. Arik
TL;DR
Text-to-SQL requires reasoning over language and complex schemas, while existing approaches can struggle with extended reasoning and sparse execution feedback. Reasoning-SQL applies GRPO with task-specific partial rewards, and reports consistently stronger accuracy and generalization than SFT, including gains over larger proprietary models on BIRD.
Problem
Existing Text-to-SQL methods can fall short on ambiguous or multi-step queries because accurate SQL generation requires substantial reasoning over language and database schemas.
Method
Reasoning-SQL uses GRPO with LLM-as-a-Judge, Syntax Check, Schema Linking, and N-Gram Similarity rewards to supplement sparse execution accuracy feedback.
Results
Reasoning-SQL models consistently outperform SFT-trained models and report gains of 4% over o3-mini and 3% over Gemini-1.5-Pro-002 on BIRD.
Takeaways & Limitations
The results support RL training with partial rewards as an effective approach for improving Text-to-SQL reasoning and generalization.
Abstract
from arXiv · showhide
Text-to-SQL is a challenging task involving multiple reasoning-intensive subtasks, including natural language understanding, database schema comprehension, and precise SQL query formulation. Existing approaches often rely on handcrafted reasoning paths with inductive biases that can limit their overall effectiveness. Motivated by the recent success of reasoning-enhanced models such as DeepSeek R1 and OpenAI o1, which effectively leverage reward-driven self-exploration to enhance reasoning capabilities and generalization, we propose a novel set of partial rewards tailored specifically for the Text-to-SQL task. Our reward set includes schema-linking, AI feedback, n-gram similarity, and syntax check, explicitly designed to address the reward sparsity issue prevalent in reinforcement learning (RL). Leveraging group relative policy optimization (GRPO), our approach explicitly encourages large language models (LLMs) to develop intrinsic reasoning skills necessary for accurate SQL query generation. With models of different sizes, we demonstrate that RL-only training with our proposed rewards consistently achieves higher accuracy and superior generalization compared to supervised fine-tuning (SFT). Remarkably, our RL-trained 14B-parameter model significantly outperforms larger proprietary models, e.g. o3-mini by 4% and Gemini-1.5-Pro-002 by 3% on the BIRD benchmark. These highlight the efficacy of our proposed RL-training framework with partial rewards for enhancing both accuracy and reasoning capabilities in Text-to-SQL tasks.
1 Introduction
Reasoning-SQL addresses the limited reasoning of existing Text-to-SQL methods with reinforcement learning that encourages detailed intermediate reasoning. Its composite partial rewards alleviate sparse execution feedback, and experiments report stronger performance than SFT and larger proprietary models.
- Existing SFT and few-shot approaches often struggle with ambiguous or multi-step queries requiring extended reasoning.
- Reasoning-SQL uses reinforcement learning to encourage detailed intermediate reasoning steps for more accurate SQL generation.
- The method combines LLM-as-a-Judge, Syntax Check, Schema Linking, and N-Gram Similarity rewards with GRPO to address sparse execution feedback.
- 4% over o3-mini and 3% over Gemini-1.5-Pro-002 are reported gains for smaller models trained with the proposed approach.
- 72.78% execution accuracy and 93% lower monetary inference cost are reported when the models are integrated into standard Text-to-SQL pipelines.
- The authors report that naturally evolved structured reasoning outperforms well-designed hand-crafted step-by-step approaches.
2 Related Works
Text-to-SQL research has progressed from rule-based systems to neural semantic parsing, prompting, and multi-stage pipelines. Recent reasoning-focused work uses reinforcement learning to extend capabilities on complex reasoning tasks, motivating this paper’s RL-based direction.
- Text-to-SQL: Early Text-to-SQL systems used handcrafted grammars, templates, and keyword matching before deep learning introduced neural semantic parsing.
- Text-to-SQL: Later systems enabled zero-shot and few-shot Text-to-SQL through prompt engineering and in-context learning.
- Text-to-SQL: Modern Text-to-SQL pipelines commonly combine schema linking, self-correction, execution feedback, and query selection to refine SQL generation.
- RL for reasoning: Reasoning-oriented language models use chain-of-thought prompting and reinforcement learning for mathematical, coding, and logical deduction tasks.
3 Methodology
Reasoning-SQL trains Text-to-SQL models with GRPO and task-specific rewards that provide granular guidance beyond binary execution accuracy. The pipeline generates candidate queries, evaluates their syntax, schema, lexical similarity, and semantic quality, then reinforces the policy.
- 3.1 Reinforcement Learning Protocol: The training framework combines GRPO with crafted rewards targeting schema linking, valid SQL syntax, and sparse execution feedback.
- 3.1 Reinforcement Learning Protocol: GRPO computes relative advantages from multiple candidates generated for each natural-language question and database schema.
- 3.2 Reward Design: Execution accuracy compares candidate and ground-truth query outputs, but its binary reward gives no credit to partially correct logic that fails exact execution.
- 3.2 Reward Design: The LLM-as-a-judge reward evaluates zero-execution-accuracy queries for logical consistency, structural similarity, and semantic correctness.
- 3.2 Reward Design: Schema Linking uses Jaccard similarity over candidate and gold schema items, while N-Gram Similarity uses Jaccard similarity over SQL n-grams.
- 3.2 Reward Design: The format reward encourages outputs with predefined <reasoning> and <answer> tags, supporting structured reasoning during training.
- 3.2 Reward Design: The final reward is a weighted sum of evaluation functions, with weights chosen so incorrect queries cannot exceed a correct query’s overall reward.The aggregation is expressed as r = ∑i w fi · r fi.
- Baselines: The study uses direct SQL-prediction SFT and bootstrapped STaR-SFT as baselines for evaluating reasoning-oriented training.
4 Experimental Setup
The experiments examine reward designs, baseline comparisons, emergent reasoning, and generalization using filtered BIRD data and multiple model sizes. The trained models are also evaluated within a standard Text-to-SQL pipeline.
- Research Questions: The experiments address reward ablations, comparisons with existing methods, emergent reasoning, and out-of-distribution generalization.
- Benchmarks and metrics: Training begins with 9,428 BIRD Question-SQL pairs from 70 databases and retains 8,026 examples after filtering noisy samples.
- Text-to-SQL pipelines: The trained models are integrated into the CHASE-SQL pipeline, replacing Gemini-1.5-Pro for fair comparison.
5 Results
Reasoning-SQL’s partial-reward GRPO training improves SQL generation over sparse execution rewards and SFT, while producing structured reasoning and strong benchmark performance. The gains extend to pipeline integration and out-of-distribution benchmarks.
- 5.1 Training Objective Ablation: 6.77% improvement over the base model exceeds the 4.11% gain from conventional SFT training alone.Adding syntax, schema, and N-gram rewards also improves their corresponding metrics.
- 5.1 Training Objective Ablation: 1.37 Syntax Check, 0.78 schema-item Jaccard, and 3.92 N-gram Jaccard gains follow addition of their associated partial rewards.Each reward directly improves its targeted metric while contributing to execution accuracy.
- 5.2 Baselines Comparison: 4% separates the RL-trained 14B model from o3-mini, while RL-trained models consistently outperform their SFT counterparts across model sizes.Table 2 reports execution accuracy with schema linking and filtered-schema execution accuracy using the correct schema.
- 5.2 Baselines Comparison: 72.29% and 72.78% execution accuracy are achieved on the BIRD development and test sets after integration into the CHASE-SQL pipeline.The integrated system uses the fine-tuned Qwen2.5-Coder-14B model for schema linking, SQL generation, self-correction, and query selection.
- 5.3 Emergent Reasoning: Structured reasoning emerges during RL training without enforcing a particular format, correcting an initially incorrect JOIN operation.By step 200, the model adopts a structured SQL-synthesis approach and identifies the join condition correctly.
- 5.4 Generalization Analysis: RL-trained models outperform state-of-the-art alternatives across Spider, Spider-DK, and Spider-Syn, whereas SFT-trained models underperform their base models.These benchmarks use distinct SQL-query and natural-language-question distributions to assess generalization.
6 Conclusion
Reasoning-SQL uses GRPO with tailored partial rewards to improve reasoning and generalization for Text-to-SQL. Across BIRD, Spider, Spider-DK, and Spider-SYN, its RL-trained models outperform SFT models, with the 14B model surpassing larger proprietary systems on BIRD.
- Reasoning-SQL applies GRPO and partial rewards to schema comprehension, query generation, and self-correction.
- RL-trained models consistently outperform SFT-trained models across BIRD, Spider, Spider-DK, and Spider-SYN.
- The results underscore the potential of partial-reward RL to improve smaller open-source models and reduce the performance gap with proprietary systems.
A.1 Training details
The training details use different optimization settings for GRPO and SFT while keeping batch size and epoch count constant. GRPO generates multiple completions per prompt to satisfy its group-size requirement.
- GRPO training uses a 1e-6 learning rate, constant scheduler, 0.1% warm-up ratio, batch size 32, and three epochs.
- Each prompt produces six completions for GRPO, while SFT uses a 1e-5 learning rate and cosine scheduler with the same batch size and three epochs.
A.2 Example Output
The appendix presents example outputs illustrating how the model’s reasoning changes during training. These examples include a GRPO-trained model output and comparisons between early and later checkpoints.
- An example from the Qwen2.5-Coder-14B all-rewards model shows step-by-step reasoning before the final SQL query.
- The model’s reasoning varies by question and adapts to the conditions required for generating the answer.
- Outputs at training step 10 are compared with reasoning outputs after one epoch to illustrate training-time changes.
A.2.1 Model Checkpoints
The checkpoint examples show the model progressively constructing SQL by identifying tables, joins, filters, and selected columns. Later outputs provide increasingly complete query formulations for the race-and-qualifying-time question.
- A.2.1 Model Checkpoints: The checkpoint outputs include progressively different SQL formulations, including queries selecting T1.code or T2.code after joining drivers and qualifying.
- A.2.1 Model Checkpoints: The checkpoint examples use a question asking for the abbreviated code of the driver with Q3 time 0:01:33 in race No. 45.
- A.2.1 Model Checkpoints: The reasoning identifies the drivers and qualifying tables, links them through driverId, and applies an INNER JOIN.
- A.2.1 Model Checkpoints: The query construction filters raceId = 45 and Q3 times using a pattern such as M:SS%, then selects the drivers table’s code column.
- A.2.1 Model Checkpoints: DISTINCT is introduced to ensure that each driver abbreviation appears only once in the result.
A.3 Prompts
The prompts guide models to generate or evaluate SQLite queries through schema understanding, question analysis, and explicit SQL-construction strategies. They include both recursive divide-and-conquer reasoning and formats for GRPO, SFT, and evaluation outputs.
- Evaluation Prompt: The evaluation prompt compares a predicted SQL query with a gold query and assigns a correctness score from 0.0 to 2.0.
- SQL Constraints: The prompts require selecting relevant columns and tables, following hints, addressing question conditions, and handling schema ambiguities carefully.
- SQL Constraints: They prescribe SQL-specific rules including joining before MAX() or MIN(), grouping before ordering distinct values, NULL handling, and STRFTIME() for date operations.
- Divide-and-Conquer Prompt: The recursive divide-and-conquer prompt decomposes complex questions into sub-questions, generates pseudo-SQL and real SQL fragments, then combines them bottom-up.
- Output Formats: The GRPO format requests detailed reasoning followed by an answer, whereas the SFT and divide-and-conquer formats require a single predicted SQL query in a specified output format.
A.4 Test-time Compute Analysis
The analysis examines model performance across question difficulties, reasoning length during training, and deployment cost. It also describes the GRPO training procedure, which samples SQL candidates and updates the policy using composite rewards.
- Performance and Reasoning Length: GRPO-trained 3B and 7B models consistently outperform STaR-SFT models across all BIRD development-set difficulty levels while using less reasoning characters.
- Cost Analysis: 1% lower performance than Gemini-1.5-pro accompanies a 93% lower cost when Qwen2.5-Coder-14B(all) replaces Gemini-1.5-pro-002 in CHASE-SQL.Estimated costs are $0.08 versus $1.25 per 1M input tokens and $0.18 versus $5.00 per 1M output tokens.
- Training Dynamics: The training analysis evaluates execution accuracy and reasoning-character counts every 10 steps through 200 steps on a 10% BIRD development-set subsample.
- GRPO Training Procedure: The GRPO algorithm samples multiple SQL candidates for each question-schema pair and computes execution, judge, syntax, schema, n-gram, and format rewards.
- GRPO Training Procedure: Candidate rewards are aggregated into advantages used by the GRPO objective to update the policy parameters.