Source-linked AI summary
Agentic-SQL Revisited: Autonomy-Based Taxonomy and Empirical Benchmark Analysis for LLM Text-to-SQL
Changruo Zhao, Zujun Peng, Yu Tian, Yuting Liu, Yiyun Su, Huiying Zhu, Luyan Zhang, Heming Zeng
TL;DR
Cross-system Text-to-SQL comparison is fragile because reported results span heterogeneous benchmarks, backbones, and inference protocols. This paper builds a provenance-tracked leaderboard organized by inference autonomy and anchors it with a Spider case study, finding uneven cross-benchmark transfer, an autonomy–cost trade-off, and difficulty-focused CoT gains.
Problem
Text-to-SQL comparisons lack an inference-time organization across heterogeneous benchmarks, backbones, and protocols.
Method
The paper aggregates author-reported metrics into a provenance-tracked leaderboard organized by a five-level, operationally measurable inference-autonomy axis.
Results
Spider gains transfer unevenly across BIRD and Spider 2.0, autonomy improves robustness at non-trivial cost, and CoT gains concentrate on Hard and Extra-Hard queries.
Takeaways & Limitations
The leaderboard, case study, and harness make cross-benchmark transfer, autonomy–cost trade-offs, reasoning-internalized regimes, and difficulty-redistributed CoT gains visible.
Takeaways & Limitations
The v1 leaderboard does not yet seriously exercise multi-turn, retrieval-augmented, enterprise-scale, or dialect-transfer protocols.
Abstract
from arXiv · showhide
LLM-based Text-to-SQL progress is reported across heterogeneous benchmarks, backbones, and inference protocols, making cross-system comparison fragile. We reframe the field as a leaderboard aggregation: we collect the metrics authors themselves report and organize them along an inference-autonomy axis spanning constrained, in-context, iterative, agentic, and reasoning-internalized generation, with traceable provenance for every cell. To anchor the aggregation empirically, we run a focused case study on Spider, comparing 8B open-source backbones with and without chain-of-thought (CoT) supervision against few-shot DeepSeek~V3 and GLM-4 baselines. Four patterns emerge: Spider gains transfer unevenly to BIRD and Spider~2.0; autonomy buys robustness at non-trivial cost; reasoning internalization sits between answer-only decoding and externally orchestrated agents; and CoT gains concentrate on Hard and Extra-Hard queries. We release a Python harness mirroring the autonomy axis so that future methods can be added directly to the leaderboard.
1 Introduction
The section argues that heterogeneous benchmarks, backbones, metrics, and inference protocols make single-number Text-to-SQL comparisons fragile. It presents an autonomy-based, provenance-tracked aggregation protocol and a Spider case study to analyze cross-benchmark transfer, inference costs, and CoT effects.
- Motivation: Single benchmark numbers are insufficient because reported metrics and inference protocols vary across Spider, BIRD, and Spider 2.0.The same paper may report execution accuracy, valid efficiency score, success rate, and latency under different backbones and protocols.
- Method: The proposed aggregation classifies systems by how much structured reasoning they externalize between the question and final SQL.It collects author-reported metrics while leaving cells blank when results are unreported or not directly comparable.
- Case study: The Spider case study compares Qwen3-8B and LLaMA3.1-8B with and without CoT traces against 3-shot DeepSeek V3 and GLM-4 baselines.It reports directly comparable EX/EM numbers across difficulty bands.
- Contributions: The paper contributes a provenance-tracked leaderboard and open-source harness covering Spider, BIRD, and Spider 2.0, alongside a difficulty-stratified Spider CoT study.The harness provides fixed dataset loaders, metric implementations, and a method-adapter interface.
2 Background
The background defines Text-to-SQL as producing executable SQL from a question, schema, and optional external knowledge, while organizing evaluation around three complementary benchmarks and metrics. The leaderboard is parameterization-agnostic and classifies systems by inference behavior rather than how model parameters were obtained.
- Task formulation: Text-to-SQL maps a natural-language question, database schema, and optional external knowledge to an executable SQL query.The schema comprises tables, columns, and foreign-key relations; external knowledge can include entity descriptions, value evidence, and domain hints.
- Task formulation: In-context systems keep model parameters fixed and vary the instruction, whereas fine-tuned systems update the parameters.The passage also notes that fine-tuning is typically performed with parameter-efficient adaptation.
- Leaderboard scope: The leaderboard admits systems based on inference behavior on held-out Spider, BIRD, or Spider 2.0 splits, regardless of parameter acquisition.This makes the benchmark parameterization-agnostic.
- Benchmark landscape: Spider tests unseen-schema generalization across disjoint databases, BIRD emphasizes large schemas, realistic values, and external knowledge, and Spider 2.0 stresses long-horizon enterprise workflows.Spider contains 8,659 training examples over 146 databases and 2,147 test examples over 40 databases, with no training-test database overlap; BIRD adds Valid Efficiency Score, while Spider 2.0 uses task-success-like evaluation.
- Evaluation metrics: The leaderboard uses Exact Match, Execution Accuracy, and Valid Efficiency Score, with each metric capturing a different aspect of SQL correctness or efficiency.EM compares normalized queries, EX compares executed result sets, and VES combines execution correctness with a runtime ratio against the gold query; Spider 2.0 additionally reports binary task success.
3 Benchmark Design
Section 3 defines inference autonomy as a measurable five-level taxonomy based on the number and structure of model calls between the input question and final SQL. This axis organizes the leaderboard so results can be sliced by autonomy level without rerunning code.
- Autonomy taxonomy: Inference autonomy measures the structured reasoning, feedback, and coordination introduced between the input question and final SQL through the number and kind of model calls.The taxonomy contains five levels: constrained single-turn, in-context single-turn, iterative refinement, agentic collaboration, and reasoning-internalized generation.
- Autonomy taxonomy: The five levels range from grammar-constrained single-pass decoding and prompt-conditioned demonstrations to fixed iterative pipelines, branching agentic controllers, and internally supervised reasoning trajectories.L0 uses one grammar-constrained forward pass; L1 uses instructions and demonstrations; L2 chains link, decompose, generate, and revise; L3 branches on intermediate outputs; L1.5 emits intermediate stages within one trajectory.
- Representative systems: The taxonomy places PICARD at L0; DAIL-SQL and few-shot DeepSeek V3 / GLM-4 at L1; DIN-SQL, DART-SQL, DTS-SQL, and TS-SQL at L2; and MAC-SQL, ExeSQL, and CHESS at L3.EllieSQL and BAP-SQL extend L3 with cost-aware and budget-aware routing, while STaR-SQL, RevDecomp-SFT, and CoT-SFT case-study rows occupy L1.5.
- Leaderboard organization: The axis uses number and structure of inference-time calls as the organizing variable, enabling autonomy-level leaderboard slices without rerunning code.Table 2 catalogs leaderboard systems by autonomy level, training paradigm, and backbone class as reported by their authors.
4 Aggregated Results
The aggregated leaderboard uses citation-traceable, directly comparable metrics, leaving uncertain or protocol-incompatible cells blank. Version 1 covers Spider and BIRD, while Spider 2.0 results are deferred until comparable success-rate reports become available.
- Aggregation protocol: Every numeric leaderboard cell maps to one source citation, and uncertain or non-comparable metrics remain blank rather than being inferred.Source values are entered as reported; metrics requiring re-derivation under another scoring protocol are excluded.
- Spider leaderboard: Table 3 aggregates Spider dev/test results, with EM on the standard test set and EX split between case-study test and literature dev results.The case-study rows use a uniform protocol on the Spider holdout test set.
- BIRD leaderboard: Table 4 aggregates BIRD dev results using EX and VES as headline metrics, while the case-study models were not evaluated on BIRD.Source-specific dev/test choices are retained when BIRD test labels are held out.
- Spider 2.0: Spider 2.0 has no directly comparable success-rate results in the catalog, so its leaderboard track is omitted from version 1.The authors plan to populate it when direct reports become available; the harness already supports per-task success-rate breakdowns.
- Case-study results: Table 5 reports difficulty-stratified Spider test-set Execution Accuracy for Easy, Medium, Hard, Extra-Hard, and All EX categories.The two 8B SFT rows differ only in whether supervision includes a five-stage reasoning trace, with the remaining protocol held identical.
5 Empirical Case Study: CoT Supervision on Spider
The Spider case study compares six configurations under one protocol and finds that fine-tuning outperforms few-shot prompting, with CoT gains concentrated on harder queries and varying by backbone. Its internally comparable results require caution when compared with literature metrics because splits, scorers, and decoding protocols may differ.
- Experimental setup: Six configurations evaluate Qwen3-8B and LLaMA3.1-8B with CoT-trace or answer-only SFT against 3-shot DeepSeek V3 and GLM-4 on Spider.All configurations use LoRA for fine-tuning and are evaluated on the full 2,147-example Spider test set.
- Findings: Every fine-tuned 8B configuration exceeds 76.0 EX, versus 51.5 EX for 3-shot DeepSeek V3 and 66.3 EX for GLM-4.On Extra-Hard, DeepSeek V3 reaches 24.0 EX, while LLaMA3.1-8B without CoT reaches 67.8 EX.
- Findings: On Qwen3-8B, CoT raises Hard EX from 70.19 to 81.09 (+10.9) and Extra-Hard EX from 71.58 to 79.25 (+7.7).Easy declines from 96.62 to 94.95 (−1.7), while Medium gains +4.1.
- Findings: On LLaMA3.1-8B, CoT leaves aggregate EX unchanged at 76.01 versus 76.01 while shifting accuracy from Hard to Extra-Hard.CoT loses 1.2 on Hard and gains 5.5 on Extra-Hard.
- Limitations: The case-study rows are directly comparable internally, but literature comparisons require matching test split, scorer, and decoding protocol.The case-study EM uses strict alias-canonicalization, whereas published RevDecomp-SFT EM of 75.4 uses another convention.
6 Cross-Method Analysis
The analysis identifies four patterns: benchmark gains transfer unevenly, greater autonomy incurs token, latency, and orchestration costs, reasoning internalization occupies an intermediate position, and CoT mainly redistributes performance across query difficulties.
- Pattern 1: Uneven benchmark transfer: 75.1 EX for PICARD trails 85.3 EX for DIN-SQL+GPT-4 on Spider, while DIN-SQL+GPT-4 reaches 55.9 EX on BIRD.Smaller reasoning-internalized SFT systems can match or exceed DIN-SQL+GPT-4 on BIRD, indicating uneven cross-benchmark transfer.
- Pattern 2: Autonomy costs: L3 systems may outperform L1 systems on BIRD, but autonomy multiplies token, latency, and orchestration costs.EllieSQL and BAP-SQL treat L1–L3 selection as a per-query routing problem, while agentic harness robustness remains open.
- Pattern 3: Reasoning internalization: L1.5 systems report Spider EX in the same range as L2 GPT-4 pipelines while using a single forward pass on an open backbone.Their supervised reasoning traces must be constructed during training, often with a stronger teacher, internalizing part of externally orchestrated reasoning.
- Pattern 4: Difficulty redistribution: On Qwen3-8B, adding CoT yields +5.2 EX overall, decomposing into +4.1 Medium, +10.9 Hard, and +7.7 Extra-Hard, with Easy slightly negative.LLaMA3.1-8B has identical aggregate EX with and without CoT: 76.0 vs. 76.0.
7 Limitations and Open Problems
The paper’s evidence base is limited by incomplete provenance, inconsistent cost reporting, and a Spider-only case study. Open problems include routing across systems, multi-turn and retrieval-aware evaluation, and broader benchmarking on BIRD and Spider 2.0.
- Provenance breadth: Blank leaderboard rows indicate missing directly comparable reports, not poor system performance; Spider 2.0 also requires more time as its scoring protocol settles.Version 1 includes only results reported under each dataset’s standard protocol.
- Cost and routing: Inference cost—tokens, latency, and dollars—is inconsistently reported, while a future routing track could evaluate per-query policies over catalogued systems.The harness already includes cost columns, but the leaderboard does not.
- Beyond single-turn SQL: Next-generation benchmarks must address multi-turn dialogue, the SQL-versus-retrieval boundary, retrieval-specific reliability failures, and enterprise privacy constraints.EHR-SeqSQL and TAG motivate broader task settings, while recent RAG studies identify context-compliance and evidence-calibration concerns.
- Case-study scope: The case study covers only Spider, two open-source backbones, and two proprietary 3-shot baselines; extending the protocol to BIRD, Spider 2.0, and more backbones would widen coverage.The paper identifies this extension as the most direct way to broaden empirical evidence.
8 Conclusion
The paper replaces survey-style organization with a provenance-tracked leaderboard benchmark organized by inference autonomy and anchored by a Spider CoT case study. Its catalog, results table, and open-source harness expose four patterns while leaving unsupported cells blank.
- Contributions: The benchmark organizes LLM Text-to-SQL results by inference autonomy rather than survey-style categories.It combines a method catalog, provenance-tracked results table, and an open-source harness whose adapter interface mirrors the autonomy axis.
- Findings: The artifacts reveal uneven cross-benchmark transfer, an autonomy–cost trade-off, a distinct reasoning-internalized regime, and CoT supervision as a difficulty-redistributor.The conclusions are anchored by a focused Spider case study of CoT supervision.
- Methodological principle: The paper and harness refuse to populate cells unsupported by source papers.They use citation-backed cells and leave blank any cells where the literature is silent.