PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentar-Scale-SQL: Advancing Text-to-SQL through Orchestrated Test-Time Scaling

Pengfei Wang, Baolin Sun, Xuemei Dong, Yaxun Dai, Hongwei Yuan, Mengdie Chu, Yingqi Gao, Xiang Qi, Peng Zhang, Ying Yan

arXiv:2509.24403v6cs.CLcs.DB

TL;DR

Text-to-SQL systems on BIRD remain behind human experts. Agentar-Scale-SQL orchestrates internal, sequential, and parallel test-time scaling; it reaches 81.67% test EX and ranks first on BIRD, though multiple LLM calls add latency.

  • Problem

    On BIRD, human experts reach 92.96% execution accuracy while the top five methods achieve around 75% on the test set.

    Source: §1 · p. 1

  • Method

    Agentar-Scale-SQL combines internal scaling through RL-enhanced intrinsic reasoning, sequential scaling through iterative refinement, and parallel scaling through diverse synthesis and tournament selection.

    Source: §1 · p. 2

  • Results

    81.67% test EX and 77.00% test R-VES establish a new BIRD SOTA, surpassing AskData + GPT-4o by 0.79% in test EX and ranking first on the leaderboard.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    The authors present the framework’s BIRD results as an effective path toward human-level accuracy.

    Source: §5 · p. 8

    Multiple LLM calls for generation, refinement, and selection add substantial computational overhead and latency, making the framework less suitable for real-time applications.

    Source: Limitations · p. 9

Abstract

from arXiv · show

State-of-the-art (SOTA) Text-to-SQL methods still lag significantly behind human experts on challenging benchmarks like BIRD. Current approaches that explore test-time scaling lack an orchestrated strategy and neglect the model's internal reasoning process. To bridge this gap, we introduce Agentar-Scale-SQL, a novel framework leveraging scalable computation to improve performance. Agentar-Scale-SQL implements an Orchestrated Test-Time Scaling strategy that synergistically combines three distinct perspectives: i) Internal Scaling via RL-enhanced Intrinsic Reasoning, ii) Sequential Scaling through Iterative Refinement, and iii) Parallel Scaling using Diverse Synthesis and Tournament Selection. Agentar-Scale-SQL is a general-purpose framework designed for easy adaptation to new databases and more powerful language models. Extensive experiments show that Agentar-Scale-SQL achieves SOTA performance on the BIRD benchmark, reaching 81.67% execution accuracy on the test set and ranking first on the official leaderboard, demonstrating an effective path toward human-level performance.

1 Introduction

Text-to-SQL aims to let people query structured databases in natural language, but BIRD results still fall short of human-expert performance. Agentar-Scale-SQL combines three test-time scaling perspectives and reports first-place BIRD leaderboard results.

  • BIRD human experts achieve 92.96% EX, while the top five methods reach around 75% on the test set.Source: §1 · p. 1
  • Prior approaches explore ensemble strategies and sequential refinement but neglect internal reasoning and an orchestrated scaling combination.Source: §1 · p. 1
  • The authors adopt the Bitter Lesson, omit complex schema-linking strategies, and argue that scaling alone can improve performance.Source: §1 · pp. 1–2
  • Agentar-Scale-SQL orchestrates internal RL-enhanced reasoning, sequential iterative refinement, and parallel diverse synthesis plus tournament selection across three stages.Source: §1 · p. 2
  • The authors describe the framework as plug-and-play across databases and project that greater compute and stronger future LLMs will lift its performance ceiling.Source: §1 · p. 2
  • The framework delineates stage-specific roles and objectives and is presented as the first to scale both SQL generation and selection.Source: §1 · p. 2
  • 81.67% test-set EX, 74.90% development-set EX, and 77.00% R-VES accompany Agentar-Scale-SQL’s first-place BIRD leaderboard ranking.Source: §1 · p. 2

2 Related Work

Related Text-to-SQL work includes prompt-based, fine-tuning-based, and hybrid approaches; recent test-time scaling methods use parallel or sequential techniques but lack an orchestrated combination of scaling dimensions.

3 Methodology

Agentar-Scale-SQL organizes test-time computation into task understanding, SQL generation scaling, and SQL selection scaling. Its pipeline retrieves context, creates and refines diverse SQL candidates, then selects a final query through pairwise tournament comparisons.

3.1 Overview

  • The framework has three stages: task understanding retrieves context, generation scaling creates SQL candidates, and selection scaling chooses a final query.Offline preprocessing prepares dual schema formats and indexes database cell values and training examples for inference.Source: §3.1 · p. 3

3.2 Task Understanding

  • Task understanding retrieves relevant database cells from question keywords and few-shot examples using a question skeleton.The two retrieval processes use embedding-based similarity over separate cell and example vector stores.Source: §3.2 · p. 4

3.3 SQL Generation Scaling

  • Two complementary generators use DDL and light schema views, then iterative refinement produces a set of SQL candidates.The reasoning generator is RL-trained, while the ICL generator uses a proprietary LLM; the refinement loop makes syntax repairs and semantic edits.Source: §3.3 · p. 4

3.3.1 Intrinsic Reasoning SQL Generator

  • The intrinsic reasoning generator uses BIRD training data alone and generates multiple SQL rollouts per input for reward-based policy updates.Its reward is based on final execution correctness and basic syntax validity.Source: §3.3.1 · p. 4

3.3.2 Diverse Synthesis

  • Diverse synthesis combines a fine-tuned reasoning generator using DDL with an ICL generator using light schema and retrieved examples.The ICL generator varies prompts, example order, models, and temperature to broaden candidate variety.Source: §3.3.2 · pp. 4–5

3.3.3 Iterative Refinement

  • Iterative refinement conditionally repairs invalid SQL syntax and revises logical flaws, refining one randomly selected query per execution-outcome group.The method groups queries by execution outcomes before selecting a query for refinement.Source: §3.3.3 · p. 5

3.4 SQL Selection Scaling; 3.4.1 Tournament Selection

  • SQL selection groups queries with identical database execution results, then runs pairwise round-robin comparisons to select the highest-scoring candidate.A reasoning selector evaluates each pair using the question, light schema, and execution results; the approach avoids relying on the most frequent answer being correct.Source: §3.4.1 · p. 5, §3.4 · p. 5

3.4.2 Intrinsic Reasoning SQL Selector

  • The SQL selector is trained with GRPO on 8.5k training-set samples and receives reward for selecting the correct query.The selector's reward is 1 for a correct selection and 0 otherwise.Source: §3.4.2 · p. 5

4 Experiments

Experiments on BIRD compare Agentar-Scale-SQL with leading methods and examine module contributions, generator complementarity, and candidate-count scaling.

4.1 Experimental Setup

  • The evaluation uses BIRD, a cross-domain benchmark with over 12,751 question-SQL pairs across 95 databases and more than 37 professional domains.Execution Accuracy is the primary metric, with R-VES used to assess generated-SQL efficiency.Source: §4.1 · p. 6

4.3 Ablation Study (RQ2)

  • Removing the reasoning generator decreases total accuracy by 4.89 points, while removing the ICL generator decreases it by 3.78 points.Without the ICL generator, challenging-question performance falls from 64.14 to 55.86.Source: §4.3 · p. 6, Table 2 · p. 8

4.2 Main Results (RQ1)

  • 81.67% test EX and 77.00% test R-VES establish a new BIRD SOTA, exceeding AskData + GPT-4o by 0.79% in test EX.The framework also outperforms Gemini-SQL by 5.54% in test EX.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 Ablation Study (RQ2)

  • Removing iterative refinement drops performance by 0.52 points, while selection scaling beats self-consistency by 1.82 points in EX.The authors attribute refinement's contribution to correcting SQL syntax and semantic errors; the most frequent execution result is not necessarily correct.Source: §4.3 · p. 7

4.4 Analysis of Generator Components (RQ3)

  • 84.29% is the combined generators’ highest upper-bound accuracy, exceeding ICL’s 81.36% and reasoning’s 75.88%.The generators uniquely solve 47 and 12 samples, respectively; reasoning performs better on simple and moderate tasks, while ICL is more effective on challenging problems.Source: §4.4 · p. 7, Figure 3 · p. 8

4.5 Impact of the Candidate Number (RQ4)

  • Increasing candidates from 1 to 16 consistently raises Pass@k across difficulty levels, with the strongest improvement on challenging queries and diminishing gains after 8.The final selection strategy reaches 74.90% accuracy with the richer, more diverse candidate pool.Source: §4.4 · p. 7, Figure 6 · p. 8, §4.5 · p. 8

5 Conclusion

Agentar-Scale-SQL combines internal, sequential, and parallel scaling to improve Text-to-SQL performance. The authors present it as an effective path toward human-level accuracy and announce future work on Exercise-Time Scaling.

5 Conclusion

  • Agentar-Scale-SQL combines internal, sequential, and parallel scaling strategies to improve Text-to-SQL performance and achieves state-of-the-art results on BIRD.The authors describe its results as an effective path toward human-level accuracy.Source: §5 · p. 8
  • The authors release code and models to support future research.Source: §5 · p. 8
  • The authors plan to develop Exercise-Time Scaling, in which agents learn through action and evolve from experience.Source: §5 · p. 8

Limitations

Agentar-Scale-SQL incurs substantial compute overhead and latency, and its performance depends on the underlying base LLM. The section also presents BIRD leaderboard comparisons for execution accuracy and reward-based valid efficiency.

Limitations

  • Multiple LLM calls for generation, refinement, and selection create substantial computational overhead and high latency, making the framework less suitable for real-time applications.The authors report that enterprise clients may tolerate several seconds of latency for complex analytics generation.Source: Limitations · p. 9
  • Performance is bounded by the base LLM’s capabilities, and failures in early stages such as task understanding can compromise the overall process.Source: Limitations · p. 9

A BIRD Leaderboard: Bar Chart and Snapshot

B Additional Results on the BIRD Test Set

The appendix includes additional results on the BIRD test set.

B Additional Results on the BIRD Test Set

  • Table 3 is identified as presenting additional results on the BIRD test set.Source: Table 3 · p. 12

C Definition of Evaluation Metrics

The appendix defines three evaluation metrics: Execution Accuracy, Reward-based Valid Efficiency Score, and Soft F1-Score. It also describes schema linking as an optional modular component rather than a built-in feature.

C Definition of Evaluation Metrics

C.1 Execution Accuracy (EX)

  • EX scores a query as correct only when its executed result set exactly matches the ground-truth result set.Differences in row or column order, mismatched values, and SQL execution errors receive a score of 0.Source: §C.1 · p. 12

C.2 Reward-based Valid Efficiency Score (R-VES)

  • R-VES measures execution efficiency only for queries that pass EX, using predicted-versus-ground-truth execution times and averaging rewards across valid queries.The evaluation uses timeouts and repeated measurements, reporting the highest score.Source: §C.2 · pp. 12, 14

C.3 Soft F1-Score

  • Soft F1-Score compares predicted and ground-truth result tables by matching row contents and aggregating cell-level true positives, false positives, and false negatives.It is insensitive to column order and robust to missing values.Source: §C.3 · p. 14

D Discussion of Schema Linking

  • The framework has no built-in BIRD schema linker, but its modular design allows one to be integrated without altering the core system.The authors note this can be useful for large-scale databases with numerous tables.Source: D Discussion of Schema Linking · p. 14

E Extended Analysis: Scaling, Efficiency, and Comparisons

The section outlines analysis of test-time scaling orchestration, scaling capabilities relative to SOTA baselines, and computational costs versus utility.

E.1 Orchestrated Test-Time Scaling

Agentar-Scale-SQL combines three scaling dimensions through designs intended to make them work together, and the authors report gains over both a strong single model and the previous SOTA. It is also presented as the first method to integrate all three dimensions.

E.1 Orchestrated Test-Time Scaling

  • 5.54% over Gemini-SQL and 0.79% over AskData + GPT-4o: the authors report that Agentar-Scale-SQL outperforms the strongest single model and previous SOTA, respectively.Diverse Synthesis combines ICL breadth with RL-tuned reasoning depth, while Tournament Selection uses an RL-enhanced pairwise judge.Source: §E.1 · p. 14

E.2 Comparison of Scaling Capabilities

  • Agentar-Scale-SQL is the first framework described as integrating internal, sequential, and parallel scaling, while most compared frameworks support only one or two dimensions.Source: §E.2 · p. 14

E.3 Computational Cost vs. Practical Utility

The section frames test-time scaling’s added latency as a deliberate trade-off, while describing cost adjustments and comparing inference calls. It also lists selected prompts and future directions.

E.3 Computational Cost vs. Practical Utility

  • Higher latency than single-pass models is presented as a deliberate trade-off for high-stakes enterprise analytics, where 10–30 seconds is often tolerable for complex queries.Source: §E.3 · p. 14
  • Users can adjust the candidate count or skip refinement to reduce costs on simpler queries.Source: §E.3 · pp. 14–15
  • Although Agentar-Scale-SQL uses more steps than single-pass models, it optimizes total LLM calls relative to Contextual-SQL while achieving SOTA performance.Table 5 compares estimated inference costs by number of LLM calls.Source: §E.3 · p. 15, Table 5 · p. 15

F Prompts

  • The selected prompts cover task understanding, reasoning generation, ICL with CoT prompting, SQL fixing, and reasoning selection.The complete prompt list is directed to the code repository.Source: F Prompts · p. 15

G Further Work

  • The authors plan to shift agent experience-building to an offline “Exercise-Time” phase, develop a fully autonomous agent, and extend the approach to other code-generation and reasoning tasks.Source: G Further Work · pp. 15–16

Related papers

Loading 2509.24403v6…