PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepEye-SQL: A Software-Engineering-Inspired Text-to-SQL Framework

Boyan Li, Chong Chen, Zhujun Xue, Yinan Mei, Yuyu Luo

arXiv:2510.17586v3cs.DB

TL;DR

Text-to-SQL has advanced, but system-level reliability remains limited and the paper identifies a lack of coherent workflow orchestration. DeepEye-SQL applies an SDLC-style process with robust schema linking, diverse SQL generation, and deterministic testing. It reports strong results across BIRD and Spider without fine-tuning, while noting that ambiguity and gold-SQL errors constrain evaluation.

  • Problem

    Text-to-SQL still lacks coherent orchestration enforcing correctness across the workflow, despite advances in individual modules and BIRD execution accuracy remaining around 70%.

    Source: §1 · p. 2

  • Method

    DeepEye-SQL reframes Text-to-SQL as an SDLC workflow, combining hybrid schema linking with relational closure, independent generators, and deterministic SQL checks.

    Source: §1 · p. 4, §3.2 · pp. 8–9

  • Results

    Without fine-tuning, DeepEye-SQL outperforms larger or extensively trained approaches across BIRD and Spider, reaching 73.5% execution accuracy on BIRD-Dev.

    Source: §6 · p. 24

  • Takeaways & Limitations

    The results suggest principled orchestration is a promising path toward reliable Text-to-SQL without relying on fine-tuning or inference scaling alone.

    Source: §6 · p. 24

    About 30% of reported failures (122 cases) are attributed to ambiguity in questions or errors in gold SQL, indicating a dataset-quality ceiling.

    Source: §4.8 · p. 23

Abstract

from arXiv · show

Large language models (LLMs) have advanced Text-to-SQL, yet existing solutions still fall short of system-level reliability. The limitation is not merely in individual modules -- e.g., schema linking, reasoning, and verification -- but more critically in the lack of structured orchestration that enforces correctness across the entire workflow. This gap motivates a paradigm shift: treating Text-to-SQL not as free-form language generation but as a software-engineering problem that demands structured, verifiable orchestration. We present DeepEye-SQL, a software-engineering-inspired framework that reframes Text-to-SQL as the development of a small software program, executed through a verifiable process guided by the Software Development Life Cycle (SDLC). DeepEye-SQL integrates four synergistic stages: it grounds user intent through robust schema linking, enforcing relational closure; enhances fault tolerance with N-version SQL generation; ensures deterministic verification via a ``Syntax-Logic-Quality'' tool-chain that intercepts errors pre-execution; and introduces confidence-aware selection that leverages execution-guided adjudication to resolve ambiguity beyond simple majority voting. Leveraging open-source MoE LLMs (~30B total, ~3B activated parameters) without any fine-tuning, DeepEye-SQL achieves 73.5% execution accuracy on BIRD-Dev, 75.07% on the official BIRD-Test leaderboard, and 89.8% on Spider-Test, outperforming state-of-the-art solutions that rely on larger models or extensive training. This highlights that principled orchestration, rather than LLM scaling alone, is key to achieving system-level reliability in Text-to-SQL.

1 Introduction

DeepEye-SQL frames Text-to-SQL as a verifiable software-engineering workflow, addressing the gap between strong module-level methods and reliable end-to-end orchestration. It combines distinct generators and deterministic checks, with confidence-aware selection and reported benchmark gains.

1 Introduction

  • Existing Text-to-SQL methods still fall short of consistent system-level reliability, with BIRD execution accuracy around 70% and further degradation in real-world deployments.The paper locates the central gap in workflow orchestration rather than individual modules alone.Source: §1 · p. 2

Our Methodology

  • Confidence-aware selection uses execution consensus to adapt review, while robust schema linking and value retrieval ground the specification before generation.The framework contrasts its deterministic lifecycle with linear or search-centric approaches that rely on single-path reasoning or probabilistic self-correction.Source: §1 · p. 3

Contributions

  • DeepEye-SQL reframes Text-to-SQL as a verifiable SDLC workflow spanning requirements, implementation, testing, and release.Its stages address intent grounding, fault-tolerant generation, deterministic verification, and confidence-aware selection.Source: §1 · pp. 3–4
  • N-version programming uses independent generators with distinct reasoning strategies instead of stochastic sampling from one generator, broadening coverage of complex queries.The paper presents this as algorithmic fault tolerance.Source: §1 · p. 4
  • The SQL Unit Testing Tool-Chain replaces probabilistic self-reflection with deterministic Syntax, Logic, and Data Quality checks that provide feedback for revision.This is intended to address confirmation bias in LLM self-correction.Source: §1 · p. 4
  • 73.5% execution accuracy on BIRD-Dev, 75.07% on held-out BIRD-Test, and 89.8% on Spider-Test are achieved with open-source MoE LLMs and no fine-tuning, outperforming larger or fine-tuned baselines.The models have ∼3B activated parameters.Source: §1 · p. 4

2 DeepEye-SQL Overview

DeepEye-SQL organizes Text-to-SQL as a four-stage pipeline inspired by the Software Development Life Cycle. Each phase maps an engineering activity to query construction and release.

Phase-1: Intent Scoping and Semantic Grounding

  • Intent scoping uses Semantic Value Retrieval and Robust Schema Linking to interpret the question and identify relevant database data, tables, and columns.The hybrid linking strategy aims to avoid a single point of failure.Source: §2 · p. 4

Phase-2: N-version Programming for SQL Generation

  • SQL generation produces parallel candidates from three independent generators, each using a distinct reasoning paradigm, rather than sampling repeatedly from one generator.The paper describes this N-version approach as enabling broader solution-space exploration, especially for complex scenarios.Source: §2 · p. 4

Phase-3: SQL Unit Testing and Revision via Tool-Chain

  • The testing phase checks each SQL candidate with specialized deterministic tools and sends detected defects to the LLM as actionable revision directives.The process externalizes verification and revision rather than relying on LLM self-correction.Source: §2 · p. 5

Phase-4: Confidence-aware SQL Selection

  • Confidence-aware selection evaluates candidate reliability from execution consensus against an acceptance criterion, releasing a result or triggering deeper comparative review.This release-stage quality gate goes beyond simply choosing the most common answer.Source: §2 · p. 5

3 The Design Details of DeepEye-SQL

DeepEye-SQL’s design grounds questions in relevant database values and combines diverse schema-linking and SQL-generation strategies in an ordered, confidence-aware workflow for verifying and selecting candidates. It also uses prompting, execution-based selection, parallelization, and conditional LLM-based revision and adjudication to limit unnecessary cost, latency, and workflow overhead.

3.1 Semantic Value Retrieval

  • Semantic value retrieval grounds generation in database values through offline indexing and online retrieval of question-relevant values.Online retrieval first extracts likely filter keywords from the question, then searches the prebuilt indices for relevant values.Source: §3.1 · pp. 6–7

3.2 Robust Schema Linking

  • Three complementary schema-linking strategies are unioned, then relational closure adds connecting key columns so the linked schema supports valid joins.The strategies are direct linking, draft-SQL-based reverse linking, and value-based linking; closure connects tables identified by the strategies through foreign-key relationships.Source: §3.2 · p. 8

3.3 N-version Programming for SQL Generation

  • Three independent SQL generators run in parallel: skeleton-based planning, masked-example in-context learning, and divide-and-conquer decomposition for complex questions.The masked retrieval strategy follows DAIL-SQL [9]; the divide-and-conquer generator recursively solves subquestions before synthesizing a query.Source: §3.3 · pp. 9–10

3.4 SQL Unit Testing and Revision via Tool-Chain

  • A deterministic Syntax →Logic →Quality checker cascade detects and guides repairs of SQL errors, with each candidate passing through the chain once.Syntax checks executability; logical checks examine joins and ordering; quality checks address ambiguous wildcards, NULL risks, and empty results. Checker directives guide targeted LLM revisions.Source: §3.4 · pp. 10–11

3.5 Confidence-aware SQL Selection

  • Execution-result clusters provide confidence scores: high-confidence leaders are selected directly, while low-confidence cases receive pairwise LLM adjudication and confidence-weighted ranking.For pairwise comparisons, the system samples three judgments and takes the majority outcome; final scores combine each candidate’s execution confidence with its adjudicated win rate.Source: §3.5 · pp. 12–13

3.6 Workflow Optimization

  • Each SQL generator uses one sophisticated prompt to perform multi-step reasoning internally, preserving structured reasoning while reducing API calls, latency, and token overhead.The prompt instructs the LLM to output only the final SQL in a single call.Source: §3.6 · p. 13
  • DeepEye-SQL parallelizes independent work—including value retrieval, three schema-linking strategies, N-version generation, and candidate revisions—to reduce end-to-end latency.Source: §3.6 · pp. 13–14
  • LLM revision runs only after a checker detects an error, while pairwise adjudication runs only in low-confidence cases.These conditions minimize unnecessary LLM invocations.Source: §3.6 · p. 14

4 Experiments

The experiments evaluate DeepEye-SQL on Text-to-SQL benchmarks, using execution accuracy as the primary metric and comparing it with fine-tuning-based and prompting-based systems. The setup also specifies benchmark protocols, implementation configurations, and a performance comparison on BIRD-Dev and Spider-Test.

4 Experiments

  • The experiments assess performance and reliability through comparative analysis, ablations, and efficiency analyses.The section introduces the experimental setup before the comparisons and component analyses.Source: §4 · p. 14

4.1 Experimental Setup

  • The setup evaluates BIRD-Dev, Spider-Test, and both Spider 2.0 subsets, Lite and Snow.BIRD and Spider test different benchmark settings; Spider 2.0 adds enterprise-oriented schemas and dialects.Source: §4.1 · p. 14
  • Execution Accuracy is primary; Spider 2.0 uses a relaxed comparison permitting redundant result columns.The setup also reports upper-bound EX for SQL candidate selection, Table/Column Recall for schema linking, and Token Cost for efficiency.Source: §4.1 · p. 14
  • Comparisons cover fine-tuning-based and training-free prompting methods, with specialized agentic frameworks added for Spider 2.0.Table 3 presents performance comparisons on BIRD-Dev and Spider-Test.Source: §4.1 · p. 14, Table 3 · p. 15
  • The implementation integrates three open-source models across two model series; Spider 2.0 experiments use DeepSeek-R1.The reported configuration also sets retrieval, sampling, and confidence-selection parameters.Source: §4.1 · p. 15

RQ1: How does DeepEye-SQL perform against existing state-of-the-art methods on challenging Text-to-SQL benchmarks?

DeepEye-SQL reports strong benchmark results and analyzes how schema linking and diverse SQL generators contribute to its performance. Its schema-linking components combine high recall with complementary recovery, while N-version generation yields a diverse candidate pool.

RQ1: How does DeepEye-SQL perform against existing state-of-the-art methods on challenging Text-to-SQL benchmarks?

  • 73.5% execution accuracy on BIRD-Dev beats XiYan-SQL (73.3%) and CHASE-SQL (73.0%); DeepEye-SQL also reaches 75.07% on BIRD-Test and 89.8% on Spider-Test.The BIRD-Dev comparison is against leading fine-tuning-based systems; the framework uses the Qwen3-Coder-30B-A3B model.Source: RQ1: How does DeepEye-SQL perform… · pp. 15–16

4.3 Robust Schema Linking Analysis

  • 98.1% table recall and 95.4% column recall accompany a 9× input-context reduction, from 5486.2 to 627.4 tokens versus using the full schema.The full Robust Schema Linking approach achieved these results on BIRD-Dev.Source: §4.3 · p. 17, Table 6 · p. 17
  • 97.0% table and 94.0% column recall for Reversed Schema Linking exceed Direct Schema Linking, with a 13.1% gain in column recall.The authors attribute the advantage to prompting for a draft query rather than explicit extraction.Source: §4.3 · p. 18
  • Value-based linking recovers trans.k_symbol by matching “household payment” to ‘SIPO,’ addressing an omission caused by the ambiguous column name.Direct and Reversed Schema Linking missed this column in the case study.Source: Figure 4 · p. 17, §4.3 · p. 18

4.4 Analysis on N-Version Programming for SQL Generation

  • 81.1% upper-bound execution accuracy for the combined generators exceeds the best individual generator’s 78.5% potential by 2.6%, indicating complementary solutions.The combined N-version module achieves 71.7% execution accuracy; 10.2% of correct answers are solved by only one or two generators.Source: §4.4 · pp. 18–19, Figure 5 · p. 18, Table 7 · p. 18

RQ5: Is the N-version programming robust to variations in few-shot example quality?

DeepEye-SQL remains stable when few-shot example quality declines, and its confidence-aware selection reviews ambiguous candidates rather than relying solely on voting. Ablations report positive contributions from every component.

RQ5: Is the N-version programming robust to variations in few-shot example quality?

  • 73.5% versus 73.4% execution accuracy for full DeepEye-SQL under Most Relevant versus Least Relevant examples, despite example quality falling from 0.58 to 0.44.The standalone ICL generator drops from 70.9% to 70.1%, while the Hybrid Generator changes from 71.7% to 71.5%.Source: RQ5: Is the N-version programming… · p. 19

4.5 Confidence-aware SQL Selection Analysis

  • 0.954 Pearson correlation links voting confidence with execution accuracy; confidence-aware selection gains 1.0% on BIRD with Gemma3-27B over consistency-based voting.The reported BIRD accuracy changes from 70.1% to 71.1%.Source: §4.5 · pp. 19–20, Table 9 · p. 20
  • 73.5% accuracy at θ_conf = 0.6, with accuracy above 73.0% throughout θ_conf ∈ [0.4, 1.0], shows stability across the tested threshold range.The threshold balances efficiency against rigorous review.Source: §4.5 · p. 20, Figure 7 · p. 20

4.6 Ablation Study

  • Removing any component degrades performance; ICL-based SQL generation has the largest individual effect at 2.5%, while Semantic Value Retrieval and the tool-chain each account for 2.1%.The authors conclude that performance reflects collaboration among modules rather than any single component.Source: §4.6 · pp. 20–21, Table 10 · p. 21

RQ9: How does DeepEye-SQL perform in terms of efficiency and scalability compared to SOTA baselines?

DeepEye-SQL reports lower token use and latency than the compared baselines, maintains accuracy across database sizes, and benefits from parallel execution on large databases. Error analysis identifies value handling as a strength and schema linking as a persistent challenge.

RQ9: How does DeepEye-SQL perform in terms of efficiency and scalability compared to SOTA baselines?

  • 17.8s average latency per query is below Alpha-SQL’s 377.1s and CHESS’s 284.4s, while DeepEye-SQL also uses 23.21K input tokens versus their 138.03K and 327.02K.The comparison reports higher accuracy for DeepEye-SQL than both baselines.Source: RQ9: How does DeepEye-SQL perform in… · p. 21
  • 3.49× speedup on Large databases reduces latency from about 80s to 23s; execution accuracy shows no significant degradation across Small, Medium, and Large schemas.Parallel execution overlaps I/O-bound stages with generation.Source: RQ9: How does DeepEye-SQL perform in… · p. 22, Figure 8 · p. 22

4.8 Error Analysis

  • On BIRD-Dev, DeepEye-SQL records only 17 Value-related errors, outperforming baselines at 56 and 39; Schema Linking remains its most challenging category, with 126 errors for DeepEye-SQL.Ground-truth-category failures account for roughly 30% of reported failures (122 cases), stemming from question ambiguity or errors in the gold SQLs.Source: §4.8 · pp. 22–23, Figure 9 · p. 23

5 Related Work

Related work frames Text-to-SQL research around fine-tuning and prompting-based approaches, then connects reliable LLM systems to established software-engineering practices.

Text-to-SQL Solutions

  • Fine-tuning adapts models using question-SQL corpora, while prompting-based methods use in-context learning and reasoning without training.The passage notes that fine-tuned systems may generalize less well to out-of-domain or highly complex scenarios, while prompting methods often rely on single generation paths and fallible self-correction.Source: §5 · p. 23

Software Engineering

  • The SDLC provides a systematic process for software creation, and reverse engineering analyzes a finished product to infer its specifications.Source: §5 · p. 23
  • N-Version Programming, unit testing, and quality gates offer fault-tolerance, component-checking, and release-control practices that the paper brings to Text-to-SQL pipelines.The passage describes systematic adoption of these principles in LLM-based pipelines as limited.Source: §5 · p. 24

6 Conclusion

DeepEye-SQL is a verifiable SDLC-style Text-to-SQL workflow targeting end-to-end correctness. Without fine-tuning, it reports strong benchmark results using open-source MoE models and suggests principled orchestration as a path toward reliability.

6 Conclusion

  • 73.5% execution accuracy on BIRD-Dev, 75.07% on the official BIRD-Test leaderboard, and 89.8% on Spider-Test, outperforming approaches relying on larger models or extensive training.The framework uses open-source MoE LLMs with about 30B total parameters and roughly 3B activated, without fine-tuning.Source: §6 · p. 24
  • DeepEye-SQL is a verifiable SDLC-style workflow for Text-to-SQL that targets system-level reliability through end-to-end correctness.Source: §6 · p. 24
  • The results suggest that principled orchestration, rather than fine-tuning or scaling up inference alone, is a promising path toward reliable Text-to-SQL.Source: §6 · p. 24

Related papers

Loading 2510.17586v3…