PaperlayerView exact revision on arXiv

Source-linked AI summary

Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL

Jiayan Lin, Yujia Liu, Zijin Hong, Zheng Yuan, Yilin Xiao, Hao Chen, Qinggang Zhang, Xiao Huang, Feiran Huang

arXiv:2608.28432v1cs.CLcs.AIcs.DB

TL;DR

Existing ICL text-to-SQL studies often report aggregate accuracy without quantifying individual design choices’ accuracy–cost contributions. This paper evaluates 17 paradigms across five modules under controlled implementations and four backbones, finding that execution-feedback refinement is a universally low-cost improvement while other modules depend on backbone conditions.

  • Problem

    Aggregate end-to-end reporting leaves the marginal accuracy and cost of individual pipeline paradigms unclear.

  • Method

    The paper re-implements 17 paradigm configurations across five modules under a unified controlled framework and evaluates them across four backbones.

  • Results

    Execution-feedback refinement improves all four primary backbones at consistently low cost, while other paradigms and resource demands vary with backbone and pipeline structure.

  • Takeaways & Limitations

    At fixed budgets, richer pipelines over mid-tier backbones are often more cost-efficient than lean pipelines on frontier backbones, and the resulting guideline transfers to five additional backbones without per-paradigm search.

  • Takeaways & Limitations

    The evaluation is conducted on the publicly available BIRD benchmark, with deployment requiring safeguards against sensitive-data exposure and harmful or incorrect queries.

Abstract

from arXiv · show

Recent advances in in-context learning (ICL) text-to-SQL have substantially improved execution accuracy on public benchmarks by assembling increasingly elaborate pipelines around the base generator, yet existing studies typically report aggregate end-to-end accuracy, without quantifying the marginal accuracy-cost contribution of individual design choices. Consequently, providing a unified, paradigm-level cost-accuracy quantification remains a critical challenge for understanding and configuring modern text-to-SQL. To address this, we instantiate 17 paradigm-level configurations across five recurring modules of the ICL text-to-SQL pipeline under a single controlled implementation, and attribute each paradigm's marginal contribution and incurred cost across all four backbones spanning diverse capability levels and reasoning styles. Our analysis reveals that execution-feedback refinement is the only paradigm whose benefit holds universally at consistently low cost, while most other modules help only under backbone-dependent conditions. Token accounting shows that input demand is more closely tied to pipeline structure, whereas output demand is more sensitive to backbone generation behavior. Cross-module analysis further shows that stacking improves accuracy on most backbones, although how the gains compose varies with backbone capability. We also find that a fixed budget is often better spent engineering a more elaborate pipeline over a mid-tier backbone than upgrading to a frontier model with a lean pipeline. These findings distill into an actionable, cost-aware tiered guideline that transfers to five additional backbones without per-paradigm search.

1 Introduction

ICL text-to-SQL pipelines improve execution accuracy by adding modules, but aggregate reporting obscures each module’s marginal value and cost. This paper addresses that gap with a controlled paradigm-level accuracy–cost analysis.

  • ICL pipelines have improved benchmark execution accuracy by assembling increasingly elaborate components around a base generator.
  • Aggregate end-to-end accuracy obscures which modules produce gains because different combinations can reach the same result.
  • Agentic refinement loops and multi-candidate sampling can consume several times, and sometimes an order of magnitude, more tokens and dollars than single-pass baselines.
  • The analysis attributes marginal accuracy and inference cost across four backbones with diverse capability levels and reasoning styles.
  • The study evaluates 17 paradigm-level configurations across five recurring modules under one unified implementation.

2 Pipeline Decomposition

The paper decomposes ICL text-to-SQL into three stages and five recurring modules, yielding 17 paradigm-level configurations that capture how representative systems combine pipeline mechanisms.

  • The pipeline comprises Pre-Processing, SQL Generation, and Post-Processing, with five modules spanning these stages.
  • Pre-Processing: Example Retrieval selects demonstrations using question, masked-question, or SQL-skeleton representations.
  • Pre-Processing: Schema Linking filters schema elements through retrieval-based, pseudo-SQL-guided, direct-LLM, or agentic strategies.
  • SQL Generation: Generation Strategy produces SQL candidates through direct generation, chain-of-thought enhancement, or question decomposition.
  • Post-Processing: Candidate Selection chooses among generated queries using execution-based voting, scoring and ranking, or pairwise comparison.
  • Post-Processing: Refinement includes execution-feedback rewriting, execution-free self-correction, and agentic revision with execution feedback.

3 Experimental Setup

The experiments use controlled single-module swaps across datasets, metrics, and backbones to isolate paradigm-level accuracy–cost differences. Shared settings remain fixed while only the target mechanism varies.

  • Datasets: The main analysis uses BIRD’s development split with 1,534 question–SQL pairs across 11 databases, while Spider supplies 1,034 samples for transfer evaluation.
  • Backbones: Four primary backbones span capability and reasoning differences: GPT-4o-mini, Gemini-2.5-Flash, DeepSeek-V4-Flash, and GPT-5.4.
  • Backbones: The study additionally tests Qwen3-235B in thinking mode, o3-mini, and five further backbones for guideline transfer without per-paradigm search.
  • Metrics: Execution accuracy is the primary metric, and oracle candidate recall upper-bounds execution accuracy achievable by selection over a fixed candidate pool.
  • Metrics: Cost per EX point measures the additional USD needed to gain one execution-accuracy percentage point over a matched reference, alongside input and output token counts.
  • Controlled Implementation: Each paradigm replaces only its target module against a global baseline, with Candidate Selection using a matched N=5 reference and other modules kept at trivial settings.

4 Single-Module Analysis

The single-module analysis identifies execution-feedback refinement as the only universally beneficial, low-cost paradigm, while other modules vary with backbone capability and reasoning style. Token costs also separate into structure-driven input demand and backbone-sensitive output demand.

  • Evaluation setup: A single-swap protocol evaluates each of 17 paradigms against a matched-budget baseline across four backbones, isolating marginal accuracy and cost.Table 1 reports execution accuracy, API cost, incremental USD per EX point, and per-question input/output tokens.
  • Universal paradigms: R1 Execution-Feedback Refinement is the only paradigm improving all four backbones at consistently low cost.It gains +1.83 to +4.89 pp, with CPP below 0.30 USD/pp throughout; rewriting is triggered only by execution errors or empty results.
  • Universal paradigms: R2 Self-Correction helps three backbones but not DeepSeek-V4-Flash, whereas R3 Agentic Refinement is broadly positive at CPP one to two orders of magnitude above R1.DeepSeek-V4-Flash shows +0.26 pp for R2 without significance, while R3 achieves higher absolute EX in several columns at much greater cost.
  • Model-dependent paradigms: Schema linking helps weaker backbones most, while example retrieval gains generally increase with baseline EX on non-reasoning backbones.S4 adds +3.19 pp on GPT-4o-mini, whereas E2 rises from +0.91 pp on GPT-4o-mini to +2.41 pp on GPT-5.4.
  • Model-dependent paradigms: External reasoning scaffolds mainly help non-reasoning backbones: G2 adds +4.24 pp on GPT-4o-mini but −0.39 pp on DeepSeek-V4-Flash, while reasoning-backbone tests show no consistent gain.Across three reasoning backbones, G2 and G3 provide no consistent accuracy improvement over direct generation despite higher cost.
  • Model-dependent paradigms: Candidate selection improves EX over the N=5 baseline on every backbone but remains constrained by the fixed-pool oracle ceiling.The best-selector gap reaches 5.74 pp on GPT-4o-mini, indicating recall limits once the candidate pool is fixed.
  • Token costs: Input demand follows pipeline structure more closely, whereas output demand varies more with backbone generation behavior.For a given paradigm, IN remains stable across four backbones, while DeepSeek-V4-Flash produces substantially more OUT than GPT-4o-mini.

5 Cross-Module and -Model Tradeoffs

The study examines how module combinations compose across backbones and whether pipeline engineering or backbone upgrading offers better cost–accuracy tradeoffs. Stacking usually improves accuracy, but interaction patterns and cost-efficient choices depend on backbone capability and reasoning behavior.

  • Module Interactions: 17 of 24 Generation × Refinement combinations outperform the better individual paradigm, with GPT-5.4 showing the strongest pattern and DeepSeek-V4-Flash the weakest.Six combinations improve over both individual paradigms on GPT-5.4, compared with one of six on DeepSeek-V4-Flash.
  • Module Interactions: On GPT-4o-mini, all six Generation × Refinement interactions are sub-additive, whereas GPT-5.4 has four super-additive interactions.Both backbones usually benefit from stacking, but their interaction terms indicate different compositional behavior.
  • Module Interactions: Generation × Example Retrieval is near-additive overall, with interaction values ranging from −0.39 to +0.20 pp across backbones.The combination beats both individual paradigms on the three non-reasoning backbones but not on DeepSeek-V4-Flash.
  • Overall Pattern: Stacking improves accuracy on most backbones, but the way gains compose varies with backbone capability, so two-paradigm results do not fully predict complete-stack performance.The study therefore evaluates backbone-specific stacks using accuracy and cost jointly.
  • Stacking vs. Upgrading: On BIRD, stacking improves EX on every backbone; at moderate budgets, Gemini-2.5-Flash’s stack Pareto-dominates GPT-5.4’s lean baseline.At high budgets, GPT-5.4’s stack has the highest EX, but its 2.29 pp advantage over Gemini-2.5-Flash’s stack costs more than 5× as much.
  • Cross-Benchmark Transfer: On Spider, unchanged BIRD-derived stacks improve EX over their lean baselines by +3.68 to +6.77 pp, preserving the backbone–pipeline tradeoff.The DeepSeek-V4-Flash and Gemini-2.5-Flash stacks both Pareto-dominate GPT-5.4’s lean baseline.

6 Cost-Efficient Configuration Guideline

The paper distills its analyses into three progressively richer configurations that use execution-feedback refinement universally and condition generation strategy on backbone reasoning behavior. The fixed guideline transfers to five additional backbones without per-paradigm search, with larger gains and costs at higher tiers.

  • From Stacks to a Tiered Guideline: Tier 1 is the bare backbone, Tier 2 adds R1 Execution-Feedback Refinement, and Tier 3 further adds E2, S2, and conditional generation.Tier 3 uses G2 for non-reasoning backbones and G1 for reasoning-capable ones.
  • From Stacks to a Tiered Guideline: The tier choices preserve components recurring across cost-efficient stacks rather than simply selecting each module’s highest-accuracy paradigm.All four stacks contain E2, S2, and R1, while generation changes with reasoning behavior.
  • From Stacks to a Tiered Guideline: R1 is the only paradigm that improves all four primary backbones at consistently low cost.This makes refinement the guideline’s universal second-tier addition.
  • Patterns Behind the Guideline: Pipeline engineering is most cost-effective on small and mid-tier non-reasoning backbones, while Tier 2 is the cost-efficient default for reasoning backbones.Tier 3 on frontier backbones is reserved for accuracy-first settings.
  • Validation on Additional Backbones: Five validation backbones improve from Tier 1 to both higher tiers without per-paradigm search, with Tier 2 gains of +1.24 to +2.61 pp and Tier 3 gains of +3.98 to +7.63 pp.Tier 2 adds limited overhead, while Tier 3 trades a larger cost increase for higher EX.

7 Related Work

Related work evaluates text-to-SQL systems from multiple angles, studies individual components and candidate selection, and increasingly combines mechanisms into elaborate inference-time pipelines. Cost-aware LLM research addresses model selection, inference resources, and workflow design.

  • Empirical Analysis of Text-to-SQL: NL2SQL360 provides multi-angle comparisons across data domains, SQL characteristics, evaluation settings, and application-oriented solution selection.The framework represents broad empirical evaluation rather than paradigm-level pipeline decomposition.
  • Empirical Analysis of Text-to-SQL: Prior component-level work studies schema linking, while other systems combine candidate generation, selection, linking, verification, refinement, and test-time scaling.Examples include CHASE-SQL, DeepEye-SQL, and Agentar-Scale-SQL.
  • Cost-Aware LLM Systems: Cost-aware LLM research spans model routing, prompt or reasoning-token reduction, budget-aware inference allocation, and workflow-level optimization.Representative approaches include FrugalGPT, SATER, RouteLLM, LLMLingua, and Sketch-of-Thought.

8 Conclusion

The study evaluates ICL text-to-SQL pipelines as modular systems, identifying which paradigms improve accuracy relative to their inference cost across backbones. It finds execution-feedback refinement broadly useful, while pipeline and backbone choices jointly shape cost-effective performance.

  • Execution-feedback refinement improves all four primary backbones at consistently low cost, making it a safe first addition.Most other paradigms pay off only under backbone-dependent conditions.
  • Input demand is closely tied to pipeline structure, whereas output demand is more sensitive to backbone generation behavior.
  • A fixed budget is often better spent engineering a richer pipeline over a mid-tier backbone than upgrading to a frontier model with a lean pipeline.
  • BIRD-derived stacks transfer to Spider, where all four backbones improve accuracy and the backbone–pipeline tradeoff persists.
  • The resulting backbone-dependent, tiered guideline transfers to five additional validation backbones without per-paradigm search.

9 Ethical Considerations

The paper evaluates text-to-SQL methods on BIRD and documents methodological choices for classifying and representing ICL paradigms. It also highlights deployment safeguards for systems that may expose sensitive data or execute harmful queries.

  • The evaluation uses the publicly available BIRD benchmark, limiting the reported empirical setting to that benchmark.
  • Deployment should enforce least-privilege access, prefer read-only execution, validate generated SQL, sandbox queries, and maintain human oversight and audit logs.
  • The taxonomy records documented paradigm use across 24 methods, with checkmarks indicating use and dashes indicating that use is not established.
  • Methods can combine multiple paradigms within a module, especially generation strategies and schema-linking signals.
  • Classification uses each paper’s primary reported system, inferring unstated paradigms from prompts or pseudocode and otherwise marking them as unestablished.

B Paradigm Implementation Details

The implementation details organize 17 controlled paradigm configurations across the three stages of the ICL text-to-SQL pipeline. Module cards document shared inputs, mechanisms, calls, fallback behavior, and cost treatment for controlled comparisons.

  • The 17 configurations are organized into pre-processing, SQL generation, and post-processing stages.Pre-processing covers Example Retrieval and Schema Linking; SQL generation covers Generation Strategy; post-processing covers Candidate Selection and Refinement.
  • Each module card specifies shared inputs, the backbone, and fallback behavior.
  • Module cards map paradigms to mechanisms and per-query LLM calls, including auxiliary calls and their cost treatment.
  • The cards identify the trivial baseline used for the main marginal analysis and controlled comparison across paradigms.

C API Pricing

The paper defines configuration costs from token charges and reports USD prices used for the analysis. These prices were recorded on May 2, 2026, with reasoning-token charges included for reasoning backbones.

  • USD costs are calculated using API prices recorded on May 2, 2026.
  • Total configuration cost sums input- and output-token charges over the full development split for each benchmark.
  • Table 8 reports API prices in USD per million tokens.For reasoning backbones, output charges include all billed reasoning tokens used in evaluation.

D Detailed Results with Confidence Intervals

The section supplements module-level results with matched-budget execution-accuracy comparisons, confidence intervals, significance tests, and cost-per-EX-point reporting. It also specifies controlled implementations for retrieval, generation, candidate selection, and the depicted preprocessing and post-processing modules.

  • Evaluation and reporting: Table 6 reports execution accuracy, changes from matched-budget reference baselines with 95% confidence intervals, and cost per EX point for every paradigm–backbone pair.The main-paper table supplies corresponding API costs and mean per-question input/output-token counts.
  • Evaluation and reporting: Comparisons use 1,534 BIRD development examples, paired-difference confidence intervals, two-sided McNemar tests at α=0.05, and N=5 or N=1 references depending on the paradigm.Candidate Selection uses an N=5 reference; all other paradigms use N=1 to isolate paradigm effects from additional candidates.
  • Pre-processing modules: Example Retrieval uses a frozen bge-large-en-v1.5 encoder, cosine-ranked normalized embeddings, leakage exclusions, duplicate removal, and up to k=3 demonstrations without demonstration schemas.Retrieved demonstrations contain the original question, optional evidence annotation, and gold SQL.
  • Generation stage: Generation Strategy supplies the question, linked or full schema, and optional evidence to a temperature-0 backbone using one sampled candidate and one generation call per query.Only parsed SQL is passed downstream; queries without parseable SQL count as incorrect.
  • Post-processing modules: Candidate Selection chooses among five temperature-0.8 direct-generation candidates and never generates a new query, while the figures document controlled preprocessing and post-processing module implementations.The preprocessing figures cover Example Retrieval and Schema Linking; the post-processing figure covers Candidate Selection and Refinement.
Loading 2608.28432v1…