PaperlayerView exact revision on arXiv

Source-linked AI summary

FlexSQL: Flexible Exploration and Execution Make Better Text-to-SQL Agents

Quang Hieu Pham, Yang He, Ping Nie, Canwen Xu, Davood Rafiei, Yuepeng Wang, Xi Ye, Jocelyn Qiaochu Chen

arXiv:2605.02815v2cs.CL

TL;DR

Text-to-SQL over large analytical databases requires navigating complex schemas, grounding decisions in actual data, and resolving ambiguous queries, while fixed pipelines limit recovery from early mistakes. FlexSQL enables database exploration throughout reasoning, generates diverse plans implemented in SQL or Python, and uses code repair with plan-level backtracking. On Spider-Snow, it achieves 65.44% Majority@8 with gpt-oss-120b, outperforming DSR-SQL with DeepSeek-R1 and ReFoRCE with gpt-o3.

  • Problem

    Large analytical databases present text-to-SQL systems with complex schemas, ambiguous queries, and the need to ground decisions in actual data, while fixed pipelines limit recovery from early mistakes.

    Source: §1 · pp. 1–2

  • Method

    FlexSQL explores the database throughout reasoning, generates diverse plans for different query interpretations, and implements each plan in SQL or Python with repair and plan-level backtracking.

    Source: §1 · p. 2

  • Results

    65.44% Majority@8 on Spider-Snow with gpt-oss-120b outperformed DSR-SQL with DeepSeek-R1 and ReFoRCE with gpt-o3.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FlexSQL outperforms strong baselines across model scales, scales effectively with increased sampling budgets, and yields consistent gains when integrated into general-purpose coding agents.

    Source: §7 · p. 10

Abstract

from arXiv · show

Text-to-SQL over large analytical databases requires navigating complex schemas, resolving ambiguous queries, and grounding decisions in actual data. Most current systems follow a fixed pipeline where schema elements are retrieved once upfront and the database is only revisited for post-hoc repair, limiting recovery from early mistakes. We present FlexSQL, a text-to-SQL agent whose core design principle is flexible database interaction: the agent can explore schema structure, inspect data values, and run verification queries at any point during reasoning. FlexSQL generates diverse execution plans to cover multiple query interpretations, implements each plan in either SQL or Python depending on the task, and uses a two-tiered repair mechanism that can backtrack from code-level errors to plan-level revisions. On Spider2-Snow, using gpt-oss-120b, FlexSQL achieves a 65.4\% score, outperforming strong open-source baselines that use stronger, larger models such as gpt-o3 and DeepSeek-R1. When integrated into a general-purpose coding agent (as skills in Claude Code), our approach yields over 10\% relative improvement on Spider2-Snow. Further analysis shows that flexible exploration and flexible execution jointly contribute to the effectiveness of our approach, highlighting flexibility as a key design principle. Our code is available at: https://github.com/StringNLPLAB/FlexSQL

1 Introduction

Text-to-SQL over large analytical databases requires navigating complex schemas, grounding decisions in data, and resolving ambiguity, while fixed pipelines limit recovery from early mistakes. FlexSQL enables database exploration throughout reasoning and combines diverse plans with flexible program generation.

1 Introduction

  • Large analytical databases can contain hundreds of tables and tens of thousands of columns, making relevant schema elements difficult to identify.In Spider2.0, nearly 10% of 152 Snowflake databases exceed 100 tables, and the largest contain 60,000 to 72,000 columns.Source: §1 · p. 1
  • Fixed pipelines retrieve schema elements upfront and revisit the database only for validation or repair, limiting recovery from wrong table choices or misunderstood columns.Downstream repair can catch syntax errors but cannot recover when the model chose the wrong tables or misunderstood a column’s meaning.Source: §1 · pp. 1–2
  • FlexSQL allows incremental schema discovery, data-value grounding, and revisions based on database findings at any point during reasoning.Its exploration tools support top-down schema navigation, inspection of actual column values, and exploratory queries throughout reasoning.Source: §1 · p. 2
  • Diversity-enforced sampling generates plans for different query interpretations, while each plan can be implemented in SQL or Python.Python supports strategies involving multi-step transformations that are cumbersome in pure SQL.Source: §1 · p. 2
  • 65.44% Majority@8 on Spider-Snow with gpt-oss-120b outperformed DSR-SQL with DeepSeek-R1 and ReFoRCE with gpt-o3, despite using a smaller backbone.Integrating FlexSQL as skills in a general-purpose coding agent yielded over 10% relative improvement on Spider2-Snow.Source: §1 · p. 2
  • A two-stage plan-to-program architecture can revise code, plans, and schema assumptions when errors are detected.The contributions also include incremental schema navigation, data inspection, and code execution throughout reasoning.Source: §1 · p. 2

2 Motivating Example

The example query over a 48-table intellectual-property database requires locating a materials-science filter and choosing among ambiguous readings of “earlier patents each one cites.” FlexSQL explores the database, verifies intermediate results, and generates varied plans for the possible readings.

Challenge 1: Schema discovery and data grounding

  • The database stores materials-science classification codes in TECH CLASS, and the filter requires inspecting values to identify codes MS-01 through MS-09.A baseline missed TECH CLASS and returned all patents filed in Q1 2014 rather than only those in materials science.Source: §2 · p. 3

Challenge 2: Query ambiguity

  • “Earlier patents each one cites” can refer to domestic citations, foreign references, or pending applications, producing different citation counts.A baseline unioned all three citation tables and overcounted by an order of magnitude.Source: §2 · p. 3

How FlexSQL handles this query

  • FlexSQL discovers TECH CLASS through schema and column inspection, then uses GetColValues to ground the materials-science filter in actual values.It also verifies the date constraint with a query against FILING INFO before writing the full query.Source: §2 · p. 3
  • The agent creates plans with different citation-table choices and date filters, implements them in SQL or Python, and uses majority voting over execution outputs.The example’s plans differ in whether they include CITATION RECORD, FOREIGN REFS, and APP REFS.Source: §2 · pp. 3–4

3 Tool Interfaces for Database Querying

FlexSQL gives agents incremental access to database schema, data values, and code execution during text-to-SQL reasoning. Agents begin with coarse metadata and retrieve further information on demand as their understanding develops.

Database hierarchy and agent inputs

  • Agents begin with the query, optional external documents, and schema names, retrieving table names, column definitions, and data values through tool calls.For databases without an explicit schema layer, navigation is flattened to proceed directly from the database to tables.Source: §3 · p. 4

Schema exploration

  • GetSchema and GetTableCol support top-down schema navigation by returning tables within a schema and columns, types, and sample values for a table.This hierarchy helps agents narrow their focus before examining individual tables.Source: §3 · p. 4

Data inspection

  • GetColValues returns distinct column values, while FindRows searches for a term and can return matching rows with values from other columns.These tools help resolve cases where schema metadata alone does not reveal the relevant categorical values.Source: §3 · p. 4

Code execution

  • SQLExecutor runs queries and returns results or errors, while PythonExecutor enables stateful multi-step data retrieval, transformation, and verification.Agents can use execution tools to test subqueries and check intermediate results before assembling a full query.Source: §3 · pp. 4–5

4 FlexSQL Framework

FlexSQL separates database-grounded plan generation from program generation, while retaining tool interaction throughout. Plans can be refined, programs repaired, and deeper plan-level errors sent back for revision before majority voting selects the final answer.

4 FlexSQL Framework

  • The framework uses Plan Generation to construct K database-grounded plans, Program Generation to implement them in SQL or Python, and majority voting to select an answer.Python solutions are transpiled to SQL when needed.Source: §4 · p. 5

4.1 Plan Generation

  • Each plan is a natural-language query strategy specifying tables, joins, and filters, grounded in the database through tool interaction.Plans cover different interpretations of the query.Source: §4.1 · p. 5
  • Before planning, FlexSQL prunes entirely null columns, groups tables with identical schemas that differ by time-based suffix, and routes to relevant schemas.Routing uses schema names, GetSchema, and GetTableCol to identify a relevant subset of schemas.Source: §4.1 · p. 5
  • Verbalized sampling generates plans in batches that differ from earlier plans, while database tools ground them in schema structure, column values, and test queries.Plan review can trigger re-exploration and regeneration when tables are missing or filters conflict with actual values.Source: §4.1 · pp. 5–6

4.2 Program Generation

  • For each plan, the agent chooses SQL or Python and tests partial implementations; code-level errors trigger program repair, while plan-level errors trigger backtracking to planning.The experiments allow up to three repair rounds for code-level errors.Source: §4.2 · p. 6
  • FlexSQL groups programs by execution output across SQL and Python, selects the largest equivalence class, and transpiles and verifies a Python solution when necessary.The transpiled query must match the consensus output before it is returned.Source: §4.2 · p. 6

5 Experiments

Experiments cover Spider2-Snow and Spider2-SQLite, analyze FlexSQL’s results and design choices, and test its integration into Claude Code.

5.1 Main results

  • 55.15% Pass@1 on Spider2-Snow and 57.78% on Spider2-SQLite yield absolute improvements of 11.03% and 9.63% over the strongest respective baselines.These results use gpt-oss-120b.Source: Table 1 · p. 7, §5.1 · p. 7
  • 65.44% Majority@16 surpasses DSR-SQL with DeepSeek-R1 (63.80%) and ReFoRCE with gpt-o3 (62.89%), while FlexSQL leads ReFoRCE across K=8–64.At K=8, FlexSQL reaches 59.74%.Source: Table 2 · p. 7, §5.1 · pp. 7–8, Figure 3 · p. 7
  • 43.20% Pass@1 from gpt-oss-20b on Spider2-Snow is comparable to the strongest 120b baseline (44.12%); on SQLite, its 50.37% beats both 120b baselines.The SQLite baselines are ReFoRCE at 45.19% and DSR-SQL at 48.15%.Source: §5.1 · p. 7

5.2 Analysis

  • On Spider2-SQLite, removing diverse planning lowers 120b Majority@8 from 64.44% to 55.56%, while disabling repair lowers it to 57.04%.Removing diverse planning raises 120b Pass@8 from 78.52% to 83.70%, showing a trade-off with majority consensus.Source: Table 3 · p. 8, §5.2 · pp. 8–9
  • Removing Python causes the largest ablation drop: Majority@8 falls from 64.44% to 52.59% for 120b and from 54.07% to 42.22% for 20b.On Spider2-SQLite, Python alone solves up to 27.6% of queries, and many solved questions have correct answers from both languages.Source: Table 3 · p. 8, §5.2 · pp. 8–9, Table 5 · p. 8
  • FlexSQL’s incremental schema exploration yields substantially higher precision, comparable recall, and the best overall F1 on Spider2-Snow.The first generated plan also attains competitive F1.Source: Table 4 · p. 8, §5.2 · p. 9
  • 96.77% transpilation accuracy was achieved on 93 questions whose majority-vote answers were produced exclusively by Python.The gpt-oss-20b pass handles about 87% of cases; gpt-oss-120b mainly addresses remaining Snowflake-specific SQL syntax.Source: Table 6 · p. 9, §5.2 · p. 9

5.3 Augmenting General-Purpose Coding Agents with FlexSQL Paradigm

  • Claude Code with the FlexSQL harness improves Spider2-Snow Pass@1 by 8.3 percentage points, from 57.7% to 66.0%, at increased average latency.Average latency rises from 216.5 to 288.5 seconds per question.Source: Table 7 · p. 9, §5.3 · p. 9

6 Related Work

FlexSQL differs from fixed-stage text-to-SQL pipelines by keeping database exploration available throughout reasoning, and applies agentic backtracking to schema and plan assumptions.

Text-to-SQL pipelines

  • Most text-to-SQL systems retrieve schema upfront and use execution feedback only for final validation; FlexSQL supports exploration and backtracking throughout reasoning.The related work describes prior extensions for planning and candidate diversity, while distinguishing FlexSQL’s flexible database interaction.Source: §6 · p. 10

Agentic reasoning

  • FlexSQL applies agentic backtracking to upstream schema and plan assumptions, not only code-level errors.This instantiates a broader strategy of revising earlier decisions based on execution feedback.Source: §6 · p. 10

Intermediate representations

  • FlexSQL uses Python as an intermediate language, allowing solutions to be implemented in Python and translated back into SQL.Prior work has used intermediate representations between natural language and SQL for generation and error correction.Source: §6 · p. 10

7 Conclusion

FlexSQL replaces fixed pipeline stages with on-demand database exploration throughout reasoning; its flexibility supports diverse planning, plan-level backtracking, and bilingual generation, with gains compounding in ablations. It outperforms strong baselines across model scales on Spider2.0, scales with larger sampling budgets, and improves performance when integrated into general-purpose coding agents.

7 Conclusion

  • FlexSQL outperforms strong baselines across model scales on Spider2.0, scales effectively with increased sampling budgets, and yields consistent gains when integrated into coding agents.The reported results use open-source models with 20B and 120B parameters; the coding-agent integration includes Claude Code.Source: §7 · p. 10

A Building Context

Before generating plans, FlexSQL compresses database metadata using three techniques to create a compact, informative representation.

A Building Context

  • FlexSQL removes columns that contain only NULL values across the dataset.Source: A Building Context · p. 15
  • Tables with identical schemas but different time suffixes are grouped together.The examples include GA SESSION tables with different date suffixes.Source: A Building Context · p. 15
  • When external documents are available, a summarization agent extracts query-relevant domain knowledge.Source: A Building Context · p. 15

B Generation ratio of Python and SQL

The section reports a breakdown of Python and SQL use among majority groups of solved questions.

B Generation ratio of Python and SQL

C Error Analysis on Python-to-SQL Transpilation

The appendix analyzes Python-to-SQL transpilation failures and illustrates where Python suits analytical tasks, alongside tool-use statistics and translation heuristics.

C Error Analysis on Python-to-SQL Transpilation

  • The transpilation failures involved cross-format parsing and selecting the second-most-frequent module, which the LLMs did not translate correctly into SQL.The formats required distinct parsing strategies, while second-place selection required ranking after counting.Source: C Error Analysis on Python-to-SQL… · p. 16
  • For the local300 recurrence task, LLMs used cumulative-sum windows instead of the clamped balance recurrence, then struggled with customer-and-month aggregation.The correct recurrence yields [5, 0, 3] for amounts [+5, −8, +3], rather than the cumulative-sum trajectory [5, −3, 0].Source: C Error Analysis on Python-to-SQL… · p. 16

D Examples of Python’s advantages on Spider2-Snow analytical questions

E Statistics of tool calls

  • GetSchema is exclusive to warehouse systems, and only 144 Spider2-Snow questions require multiple schemas, accounting for its relatively low tool-call count.Source: E Statistics of tool calls · p. 17

F Heuristic rules for transpilation

  • The transpilation heuristics preserve runtime types and distinguish identifiers, strings, and Python–Snowflake regex semantics.For case-insensitive REGEXP_LIKE, the guidance uses the flag ’is’; VARIANT access also requires full paths through nested keys and array indices.Source: F Heuristic rules for transpilation · pp. 17–18
  • For row-wise Python helpers that parse JSON or VARIANT values into lists, the rules translate unnesting through Snowflake’s LATERAL FLATTEN.Source: F Heuristic rules for transpilation · p. 17

G Inference Configurations

The paper reports its inference configuration, explains its focus on Spider2.0, and limits baseline comparisons to open-source approaches with publicly available code and literature.

G Inference Configurations

  • Experiments used local vLLM instances with at most four Nvidia L40s GPUs, high reasoning effort, and temperature 1.0.These instances served gpt-oss models.Source: G Inference Configurations · p. 18

H Additional discussions

  • The evaluation focuses on Spider2.0 because its massive schemas and dialect-specific features stress-test advanced reasoning and planning.The authors describe standard Spider as abstracting away many real-world challenges.Source: H Additional discussions · p. 18
  • Baselines are restricted to open-source approaches with public code and literature; proprietary leaderboard entries are excluded because their systems and compute budgets lack transparency.The authors say this limits fair baseline comparisons.Source: H Additional discussions · p. 18

Related papers

Loading 2605.02815v2…