PaperlayerView exact revision on arXiv

Source-linked AI summary

Iteration Without Elaboration: A Simple ReAct Architecture Suffices for Text-to-SQL Generation

Jian Lu, Haiwei Yu, Raymond M Xiong, Anru Zhang, Danyang Zhuo

arXiv:2608.22651v1cs.CL

TL;DR

Existing text-to-SQL pipelines add components that increase latency and engineering overhead. ReAct-SQL instead uses a typed DSL with a simple ReAct loop and execution feedback, achieving competitive or leading accuracy on two benchmarks while remaining faster than elaborate baselines.

  • Problem

    Existing text-to-SQL pipelines rely on added modules and stages that increase latency and engineering overhead, limiting interactive use.

  • Method

    ReAct-SQL combines a typed DSL of relational operations with a think–act–observe loop that uses compiled-SQL execution feedback to construct queries iteratively.

  • Results

    ReAct-SQL reaches 84.5% accuracy on corrected BIRD mini-dev and 73.9% on EHR-SQL, while matching or exceeding elaborate baselines and achieving substantially lower latency.

  • Takeaways & Limitations

    A plain ReAct loop can deliver competitive text-to-SQL performance at lower cost and latency, while iteration addresses grounding and the DSL improves compositional reliability.

  • Takeaways & Limitations

    For enterprise databases with thousands of tables, explicit schema pruning remains necessary to fit the schema within the model’s context window.

Abstract

from arXiv · show

Modern text-to-SQL systems have become increasingly elaborate, relying on schema-linking modules, retrieval-augmented prompting, candidate generation, and multi-stage refinement pipelines. While effective, these additions introduce substantial latency and engineering overhead. To this end, we present \textbf{ReAct-SQL}, a simple yet effective zero-shot ReAct-style framework built solely on iterative reasoning and a constrained action space defined by a typed Domain-Specific Language (DSL) of 15 relational operations, rather than free-form SQL generation. The model incrementally issues DSL calls, observes compiled-SQL execution feedback, and revises its reasoning through interaction. On corrected BIRD mini-dev and EHR-SQL, ReAct-SQL achieves \textbf{84.5\%} and \textbf{73.9\%} accuracy, respectively, matching substantially more elaborate baselines while running up to $8\times$ faster. Incremental ablations further show that iteration primarily improves grounding, while the DSL improves compositional reliability.

1 Introduction

Text-to-SQL is a schema-conditioned reasoning problem whose elaborate solutions add cost and latency. ReAct-SQL addresses composition with a typed DSL and grounding with iteration, achieving competitive accuracy with a simpler pipeline.

  • Motivation: Text-to-SQL correctness depends on the target database schema, making it a schema-conditioned reasoning problem rather than free-standing code generation.Different schemas can require different queries, while incorrect columns, joins, or key formats can silently invalidate answers.
  • Motivation: Existing approaches add components for incremental accuracy gains but often underweight their cost-latency profile and retrieval requirements.High latency limits interactive use, while retrieved question–SQL pairs require labeled corpora that may be unavailable for private databases.
  • Design Rationale: Composition and grounding are distinct failure modes: typed operations constrain executable query construction, while iteration supports schema and value disambiguation.Iteration can probe value distributions, verify join keys, and sample rows before finalizing a query.
  • Proposed Framework: ReAct-SQL is a zero-shot framework that constructs queries through a typed DSL and ReAct-style loop without direct SQL generation, schema linking, retrieval, or candidate prefilling.The resulting trace also records step-decomposed reasoning.
  • Contributions: ReAct-SQL’s vanilla think→act→observe loop reaches competitive accuracy at substantially lower cost and latency than more elaborate pipelines.The framework uses a constrained action space of 15 schema-validated relational operations.
  • Contributions: The paper evaluates text-to-SQL systems end-to-end under identical conditions using accuracy, cost, and latency as first-class metrics.The analysis targets deployable system performance rather than accuracy alone.

2 Related Work

Related work includes one-pass prompting, fixed multi-stage pipelines, ReAct-style agents, and structured generation. ReAct-SQL combines iterative tool use with a small typed relational DSL as its entire action space.

  • Text-to-SQL Generation: Early text-to-SQL systems used sequence-to-sequence models, while schema-aware in-context prompting offered a one-pass alternative that proved brittle on complex schemas.Later multi-stage pipelines decompose the task into fixed sequential steps targeting different sub-problems.
  • Text-to-SQL Generation: Multi-stage systems coordinate specialized components such as retrieval, schema selection, candidate generation, and unit testing.CHESS exemplifies a fixed-order pipeline with four specialized agents.
  • LLM Agents and ReAct-Style Reasoning: ReAct interleaves reasoning traces with tool-calling actions, and later extensions add self-reflection, learned tool use, or persistent skill libraries.ReAct-SQL applies the basic interaction pattern to text-to-SQL.
  • Structured and DSL-Based Generation: Structured generation improves reliability either by constraining SQL tokens or by replacing SQL with an executable action language.Examples include PICARD, Outlines, and CodeAct.
  • Structured and DSL-Based Generation: ReAct-SQL defines a closed action space of 15 typed relational operations rather than constraining SQL tokens or reusing a general-purpose language.The DSL constitutes the entire action space.

3 Methodology

ReAct-SQL incrementally constructs queries through a think→act→observe loop over typed Relation operations. A deterministic engine validates and compiles calls, while execution feedback supports revision and final answer shaping.

  • Think–Act–Observe Loop: The model receives the schema, question, and DSL documentation, then repeatedly thinks, issues batched DSL operations, observes results, and revises its query.The loop terminates when the model emits a final signal subject to a configurable turn budget.
  • Think–Act–Observe Loop: A deterministic engine validates typed DSL calls against the schema and Relation graph before compiling them to SQL.Typed errors identify failed calls and allow the next reasoning turn to reissue corrections.
  • Think–Act–Observe Loop: Execution observations include output handles, available columns, result previews, and row counts, which condition later reasoning turns.Handles persist across turns, allowing the model to reuse prior intermediate results.
  • Query Construction via Relation Operations: Each immutable Relation represents a SQL subquery and available columns, so operations create a DAG that the engine compiles into SQL.Logical column names insulate the model from SQL-level scoping rules.
  • Query Construction via Relation Operations: The DSL contains 12 Relation-building operations and 3 read-only probing operations for query construction, value exploration, and format verification.Probing operations include probe_values, value_exists, and sim_value_in.
  • Query Construction via Relation Operations: Well-defined column transformations let the engine catch unknown, stale, or dropped-column references at compile time rather than execution.Operations such as joins and aggregates update available columns according to SQL semantics.
  • Answer Shaping: An answer-shaping pass either selects a canonical execute result with a sentinel or synthesizes a combined table from multiple executions.This pass presents the final answer without further reasoning.

4 Experimental Setup

The evaluation uses corrected BIRD mini-dev and EHR-SQL under controlled comparisons with multiple text-to-SQL baselines. It also highlights benchmark correction, domain shift, exact-result scoring, and retrieval constraints.

  • Benchmarks: Corrected BIRD mini-dev contains 498 questions across 11 databases with per-question difficulty labels.An audit found annotation errors in the original release that can markedly shift system accuracy and ordering.
  • Benchmarks: EHR-SQL tests domain shift using 934 answerable clinical questions over approximately 15 interconnected medical tables.Its labeled training split is withheld, preventing in-context retrieval from question–SQL pairs.
  • Evaluation Protocol: The official BIRD protocol requires exact equality between predicted and gold result sets, penalizing numerical drift, extra columns, and row-order differences.The passage notes analogous mismatches in EHR-SQL.
  • Baselines: The comparison spans CHESS, RSL-SQL, DSR-SQL, DIN-SQL, and MAC-SQL alongside ReAct-SQL.The systems are compared under controlled conditions using the same model family.

5 Results

ReAct-SQL remains competitive across BIRD and EHR-SQL while substantially reducing latency and occupying the cost–latency Pareto frontier. Its performance advantage is especially clear on EHR-SQL and persists across tested model backbones.

  • Corrected BIRD mini-dev: 84.5% accuracy on corrected BIRD mini-dev places ReAct-SQL within 1.2 points of CHESS, DSR-SQL, and RSL-SQL.ReAct-SQL is the fastest system on BIRD, with 7.5 s median latency.
  • Corrected BIRD mini-dev: 7.5 s median latency makes ReAct-SQL 7.2× faster than CHESS and 3.8× faster than DSR-SQL on BIRD.It matches the accuracy of both systems through a single ReAct loop without their candidate sampling or schema-refinement scaffolding.
  • EHR-SQL: 73.9% accuracy makes ReAct-SQL the leader on EHR-SQL, 3.8 points above CHESS and 4.7 above DSR-SQL.Its 10.6-point drop from BIRD is the smallest among the six systems.
  • Efficiency: ReAct-SQL lies on the Pareto frontier in all four cost–latency panels, with no system simultaneously cheaper or faster at equal accuracy.RSL-SQL is cheaper on BIRD but has lower accuracy.
  • Cross-backbone consistency: ReAct-SQL improves over the zero-shot raw-SQL baseline on every tested backbone, with gains ranging from +15.0 to +36.5 points.The evaluation uses a 200-question EHR-SQL subset spanning four backbones from two model families.

6 Ablation Study

The ablation separates ReAct-SQL’s two design choices: iteration primarily addresses grounding, while the typed DSL improves compositional reliability. Their relative benefits reverse across benchmarks, but combining both performs best on each.

  • Result overview: 78.5% baseline accuracy on BIRD contrasts with 45.9% on EHR-SQL, revealing sharply different benchmark difficulty profiles.The two benchmarks begin with a 33-point gap under zero-shot raw SQL.
  • Result overview: 45.9% to 72.2%: iteration alone raises EHR-SQL accuracy by 26.2 points, while adding the DSL contributes only 1.7 additional points.The iteration gain is highly significant (McNemar p < 10^-45).
  • Result overview: +0.6 points from iteration versus +5.4 points from the DSL on BIRD reverses the EHR-SQL pattern.The DSL contribution is statistically significant on BIRD (McNemar p = 0.006).
  • Two bottlenecks, two remedies: Iteration probes database values and conventions to resolve grounding uncertainty in specialized clinical schemas.The loop uses operations such as probe_values, value_exists, and sim_value_in.
  • Two bottlenecks, two remedies: The typed DSL abstracts SQL grammar into validated relational steps, helping prevent structural errors in deeply nested or multi-join queries.Compile-time validation and materialize address mistakes that raw-SQL ReAct may need to rewrite from scratch.
  • Conclusion: Iteration addresses the grounding bottleneck, the DSL addresses the composition bottleneck, and only their combination delivers the best performance on both benchmarks.The dominant bottleneck shifts between EHR-SQL and BIRD.

7 Analysis and Implications

The analysis argues that iterative interaction can subsume much of the grounding machinery used by elaborate text-to-SQL pipelines, while a constrained DSL improves reliability for compositional complexity. It also frames efficiency as a first-class deployment concern because strong systems can have similar accuracy but very different costs and latency.

  • Iteration subsumes the grounding pipeline: Static schema-linking, retrieval, and exploration stages can be replaced by model-directed database inspection during generation.These prior mechanisms provide database knowledge before generation, whereas iteration lets the model resolve uncertainty interactively.
  • Iteration subsumes the grounding pipeline: 45.9% to 72.2%: a ReAct loop without schema linking, retrieval, or exploration modules improves EHR-SQL accuracy by 26.2 points.The loop folds database inspection into the model rather than using a fixed preprocessing stack.
  • A constrained action space yields reliable SQL generation: A typed DSL exposes a closed set of relational operations, allowing the model to assemble complex queries without emitting raw SQL grammar.The design targets structurally complex queries involving nested subqueries, multi-way joins, and layered aggregation.
  • Why this matters: accuracy has saturated, efficiency has not: Accuracy has largely converged among strong BIRD systems, while latency spans nearly an order of magnitude.Added retrieval, candidate sampling, verification, and orchestration components create cost and latency differences even within a similar accuracy band.
  • Why this matters: accuracy has saturated, efficiency has not: Deployable text-to-SQL evaluation should treat accuracy, cost, and latency as first-class metrics rather than accuracy alone.The analysis identifies efficiency as the main axis separating systems with comparable BIRD accuracy.

8 Conclusion

ReAct-SQL uses a minimal combination of a typed DSL with compile-time validation and a ReAct loop with execution feedback. It achieves competitive accuracy while reducing cost and latency, and the ablation links iteration to grounding and the DSL to composition.

  • Conclusion: ReAct-SQL combines a typed DSL with compile-time validation and a ReAct loop with execution feedback.The framework is deliberately minimal and zero-shot.
  • Conclusion: Up to 8× faster: ReAct-SQL reaches competitive accuracy on corrected BIRD mini-dev and EHR-SQL while reducing efficiency costs.The conclusion frames the result across both benchmarks rather than a single dataset.
  • Conclusion: Iteration addresses grounding, whereas the DSL addresses composition, separating two orthogonal bottlenecks in the ablation.The DSL also decomposes reasoning into step-by-step relational operations, while the loop subsumes specialized preprocessing stages.
  • Conclusion: Pipelines within a few accuracy points can differ by more than an order of magnitude in cost and latency.The paper therefore advocates evaluating the full accuracy–cost–latency trade-off.

Limitations

The approach depends strongly on the base model’s ability to follow structured instructions, and its current evaluation does not establish performance for enterprise-scale schemas or DSL completeness.

  • Model dependence: Weaker models may hallucinate nonexistent operations or emit malformed DSL calls, substantially degrading performance.This limitation follows from ReAct-SQL’s reliance on structured instruction following.
  • Deployment scope: Enterprise databases with thousands of tables may still require explicit schema pruning to fit the schema within the model’s context window.The tested schemas are moderately sized, where interactive exploration successfully subsumes explicit schema linking.
  • DSL expressiveness: The DSL was empirically sufficient in these experiments, but its formal completeness over standard relational operations remains unproven.The authors observed no failure attributable to inexpressibility while leaving a rigorous proof for future work.

Ethical considerations

The work relies on publicly available research artifacts and uses them for their intended evaluation purposes, while the clinical data requires credentialed access and responsible-use safeguards.

  • The BIRD and EHR-SQL benchmarks are publicly released for research use and are used solely to evaluate text-to-SQL systems.
  • The compared baseline systems are run from their authors’ public code releases.
  • The EHR-SQL experiments use a de-identified MIMIC-IV port requiring credentialed access, human-subjects training, and a data use agreement.
  • The clinical data is used strictly for non-commercial research, with no attempt to re-identify individuals and no new human data collected.

A Statistical Analysis

The paper supports its ablation and difficulty analyses with paired statistical tests, bootstrap confidence intervals, and benchmark accuracy breakdowns across official difficulty labels.

  • Statistical tests: McNemar’s paired test evaluates each pairwise accuracy difference using shared questions and discordant correctness pairs.For BIRD, continuity-corrected χ2 p-values are reported; selected extreme-tail EHR-SQL comparisons use exact-binomial p-values.
  • Ablation analysis: Table 5 reports ∆Acc, discordant-pair counts b and c, McNemar p-values, and paired bootstrap 95% confidence intervals for the ablation comparisons.
  • Difficulty analysis: 87.2% →82.4% accuracy from simple to challenging BIRD questions corresponds to a 4.8-point drop, while reasoning turns increase from 2.6 to 4.0.All difficulty strata remain below the 15-turn budget.

C Backbone Generalization

Across backbones, ReAct-SQL improves over zero-shot raw-SQL baselines, while its trace illustrates grounding through probing, DSL-based composition, execution feedback, and inspectable intermediate results.

  • Backbone generalization: ReAct-SQL improves over the zero-shot baseline on every tested backbone, with gains ranging from +15.0 points to +36.5 points.
  • Backbone generalization: Claude Sonnet 4.6 reaches 87.0% Full accuracy, while its +34.0 gain is slightly smaller than Pro’s because its zero-shot baseline is already 53.0%.
  • Scope: The 200-question EHR-SQL subset may differ modestly from the full 934-question results because of sampling variability.The appendix uses the qualitative cross-backbone comparison as its target.
  • Efficiency mechanisms: Batched tool calls can collapse a typical query into 3–4 round-trips instead of one LLM round-trip per operation.Up to 8 tool calls may be emitted per turn, with earlier call outputs visible to later calls.
  • Execution controls: Materialization resets nesting depth by persisting a Relation to a temporary table, while derive_column flattening avoids unnecessary subquery layers.
  • Worked trace: The worked trace grounds filters by probing stored Currency and Date values before committing to expressions.
  • Worked trace: The trace terminates after arithmetic on a prior execution result, returning the scalar 402,524,570.17.
  • Worked trace: The typed DSL composes the query through intermediate handles, while iteration probes the database and execution feedback supports revision.Each intermediate handle is independently inspectable.
Loading 2608.22651v1…