PaperlayerView exact revision on arXiv

Source-linked AI summary

A Cost-Aware Agentic Architecture for NL-to-SQL over Nested Enterprise Schemas, with a New Benchmark

Yoga Sri Varshan Varadharajan, Ajay Yadav, Ritesh Goru, Prateek Chaudhury, Constantine Caramanis, Prateek Jain, Divyateja Pasupuleti, Sunil Kumar Pandey

arXiv:2609.04641v1cs.AI

TL;DR

Existing NL-to-SQL benchmarks do not characterize the nested, graph-like schemas encountered in enterprise deployments. This paper pairs the DevRev benchmark with a cost-aware single-generation architecture using iterative schema selection, metadata-rich retrieval, and structured error handling. The system achieves 91.7% Answer Correctness on DevRev, outperforming FlexSQL by 54.6 percentage points, at $0.57 per correctly answered query.

  • Problem

    Existing NL-to-SQL benchmarks do not characterize the graph-like, semi-structured, nested schemas encountered when deploying systems on enterprise data.

    Source: §1 · p. 1

  • Method

    The paper pairs the DevRev benchmark with a cost-aware single-generation architecture combining iterative schema selection, sub-field-aware metadata retrieval, and structured error handling.

    Source: §6 · p. 8

  • Results

    91.7% Answer Correctness on DevRev exceeds FlexSQL’s 37.1% by 54.6 pp, at $0.57 per correctly answered query.

    Source: §5.2 · p. 7

  • Takeaways & Limitations

    Detailed retrieval context matters for accuracy, with metadata removal showing the largest reported ablation effect, especially on high-D7 queries.

    Source: §5.5 · p. 8

    The architecture requires an execution API for iterative error feedback and therefore cannot operate offline.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Natural-language-to-SQL systems have ad- vanced rapidly on academic benchmarks, yet production enterprise schemas exhibit graph- like, semi-structured, deeply nested structure that current benchmarks do not measure. We make two complementary contributions. First, we introduce the DevRev NL2SQL bench- mark: 900 execution-verified queries with nested-type and link-graph structure, accom- panied by the Semantic Depth Score (SDS), a schema-agnostic rubric for analytical reasoning depth. Second, we present a cost-aware single- generation agentic architecture whose schema- selection, metadata-retrieval, and error-repair components are designed for the requirements this regime imposes. On the DevRev NL2SQL benchmark the system attains 91.7% answer correctness, a margin of 54.6 percentage points over the next-best baseline; on the Spider 2.0 Snowflake public dataset, it is competitive with leading systems at a single-generation operating point.

1 Introduction

Nested, link-graph enterprise schemas pose NL2SQL challenges largely unmeasured by existing benchmarks. The paper introduces a 900-query DevRev benchmark and a cost-aware, single-generation agentic architecture addressing schema discovery, nested-field metadata, and structured error repair.

1 Introduction

  • Enterprise workflow schemas combine typed ARRAY[STRUCT] columns, polymorphic foreign keys through typed link arrays, and deeply nested objects.Analytical questions require reasoning over array elements and traversing typed link graphs, with SQL combining LATERAL FLATTEN, conditional aggregation, and enum-constrained filters.Source: §1 · p. 1
  • 5% of Spider 2.0-Snow tables carry table-level descriptions, while its flat schemas and essentially absent nested-traversal queries leave important enterprise characteristics unmeasured.Its 7,860 tables across 152 databases use primitive-typed columns; DevRev fields instead carry English descriptions documenting operational intent. The paper identifies this schema-as-documentation property and the semi-structured, link-graph regime as benchmark gaps.Source: §1 · p. 1
  • Feedback-driven schema discovery, metadata-rich retrieval, and structured error repair address graph-like schemas and nested-type SQL failures.An LLM browses a precomputed knowledge graph through tools, revisiting discovery after schema-coverage errors; correct ARRAY[STRUCT] flattening requires sub-field schemas and representative sample values. A taxonomy of 15+ error classes maps execution failures to targeted repair directives.Source: §1 · pp. 1–2
  • Longer schemas and SQL, slower nested-type executions, and retry-triggering failures raise enterprise inference costs, motivating a single-generation architecture.Many leading Spider 2.0 systems use tournament-style candidate selection (Pourreza et al., 2025; Gao et al., 2024b; Wang et al., 2025). The proposed approach could nevertheless be composed with multi-candidate generation.Source: §1 · pp. 1–2
  • 900 execution-verified queries constitute the planned DevRev NL2SQL release, grounded in a production Snowflake schema with 30 core entity types and 190 total schema variants.The schema contains 13,898 fields, including 2,036 typed ARRAY[STRUCT] columns and 1,412 STRUCT-typed or nested fields, with up to four nesting levels.Source: §1 · p. 2
  • 91.7% correctness on the full 900-query DevRev corpus and 91.5% on the high-SDS subset (SDS ≥11) demonstrate the system’s benchmark performance.The system is also competitive with leading systems on the Spider 2.0 Snowflake public leaderboard at a single-generation operating point and requires no fine-tuning.Source: §1 · p. 2
  • Additional safeguards combine pre-execution plan verification, deterministic checking and rewriting, a dynamic cheatsheet (Suzgun et al., 2026), and semantic result validation.The verifier audits structural flaws, while deterministic rewriters fix cross-dialect functions and identifier casing. The cheatsheet accumulates transferable SQL rules across queries, and the semantic validator rejects clear contradictions between executed results and the question.Source: §1 · p. 2

2 Related Work

The paper positions its agentic workflow around iterative schema expansion and structured error handling. It distinguishes DevRev from existing benchmarks and synthetic-data frameworks through its focus on evaluation, analytical reasoning depth, and nested-type, link-graph schemas.

Multi-Candidate and Test-Time Scaling

  • Tournament-style NL2SQL systems combine multiple candidates with preference optimization, ensembling, sequential refinement, or execution-guided selection.CHASE-SQL (Pourreza et al., 2025) combines multi-path reasoning with preference-optimized selection, while XiYan-SQL (Gao et al., 2024b) ensembles generators using diverse prompts. Agentar-Scale-SQL (Wang et al., 2025) combines parallel synthesis with sequential refinement; DeepEye-SQL (Li et al., 2025) pairs N-version generation with execution-guided, confidence-aware selection.Source: §2 · p. 2

Agentic and Tool-Augmented NL2SQL

  • The two-tier orchestrator separates workflow control from SQL synthesis, emphasizing iterative schema expansion and a structured error taxonomy.MAC-SQL (Wang et al., 2023) uses collaborating decomposer, refiner, and selector agents; DIN-SQL (Pourreza and Rafiei, 2023) combines task decomposition, in-context learning, and self-correction. DAIL-SQL (Gao et al., 2024a) systematizes generation strategies, while AskData (Shkapenyuk et al., 2025) enriches enterprise schema context through offline metadata extraction.Source: §2 · p. 2

NL2SQL Benchmarks

  • DevRev complements Spider 2.0-Snow by testing nested-type traversal and link-graph reasoning on one production schema rather than schema linking across many flat tables.Spider 2.0 (Lei et al., 2025) includes a Snowflake subset with 7,860 tables across 152 databases, but no declared sub-field schemas or typed link arrays. Spider 1.0 established cross-domain evaluation with 200 moderately complex databases and primitive-typed columns; BIRD added messier schemas and external-knowledge questions while retaining flat tables.Source: §2 · p. 2

Synthetic NL query generation

  • DevRev generates evaluation queries rather than fine-tuning data, using SDS dimension targets, six-persona rewriting, and nested-type, link-graph schemas.SING-SQL (Cafero˘glu et al., 2025) generates in-domain training data by hierarchically partitioning flat databases into sub-schemas and synthesizing SQL across complexity tiers. DevRev instead prioritizes analytical reasoning depth and introduces lexical variation across business roles.Source: §2 · pp. 2–3

3 The DevRev Benchmark

The DevRev benchmark targets nested-type traversal and link-graph reasoning in enterprise schemas through a four-stage query-generation pipeline. Its Semantic Depth Score (SDS) measures analytical reasoning separately from SQL syntactic complexity.

3.1 The Enterprise Schema Regime

  • DevRev targets nested-type traversal and link-graph reasoning over one production schema, contrasting with Spider 2.0-Snow’s large-scale schema linking over flat tables.Spider 2.0-Snow has no STRUCT columns; its VARIANT columns lack DDL sub-field schemas. Enterprise schemas instead expose ARRAY[STRUCT] signatures, nesting up to four levels, and operational field descriptions, requiring LATERAL FLATTEN, enum-constrained filtering, and multi-hop typed-link traversal.Source: §3.1 · p. 3

3.2 Schema Overview

  • 15 of 30 core entity types underpin the benchmark, selected for available detailed m-schema documentation from a database with 190 schema variants.Polymorphic link arrays carry type and display_id fields; resolving join targets requires type filtering and an external entity mapping. Non-link ARRAY[STRUCT] fields—including surveys_aggregation, tags, and owned_by—require LATERAL FLATTEN with entity-specific sub-field signatures.Source: §3.2 · p. 3

3.3 Four-Stage Generation Pipeline

  • The generation pipeline enforces analytical depth through Column Planning, Formula Retrieval, Query Generation, and Persona Rewriting.These stages select column subsets, derive structural patterns, generate questions using DevRev-specific columns and business concepts, and rewrite each question into a distinct business-role voice.Source: §3.3 · p. 3

3.4 The Semantic Depth Score (SDS)

  • 0–5 scores across seven dimensions give SDS a maximum of 35, measuring analytical reasoning independently of SQL syntactic complexity.The dimensions are D1 Eligibility, D2 Derived Metric, D3 Population Scoping, D4 Multi-Stage Logic, D5 Output Precision, D6 Temporal Precision, and D7 SQL Construct Diversity.Source: §3.4 · pp. 3–4
  • SDS separates semantic depth from clause counts and supports targeted diagnosis of temporal-reasoning versus advanced-construct failures.Deeply chained derivations can score highly despite few clauses, while long SQL containing simple aggregations can score low on derived metrics and multi-stage logic. High-D6 and high-D7 queries isolate temporal and advanced-construct weaknesses, respectively.Source: §3.4 · p. 4

4 System Design for Nested Enterprise Schemas

The system separates workflow orchestration from single-query SQL generation, using iterative schema discovery and history-aware repair for nested enterprise schemas. Pre-execution verification, deterministic checks, persistent SQL rules, and post-execution semantic validation support the query lifecycle.

4.1 Design Principles from the Regime

  • The design combines iterative schema selection, sub-field-aware retrieval, structured history-aware repair, and a single-generation operating point for costly enterprise queries.Polymorphic typed-link arrays make required tables unpredictable before generation, while ARRAY[STRUCT] columns require nested schemas, enum vocabularies, and sample values beyond DDL signatures. These mechanisms reduce variance otherwise absorbed by multi-candidate generation.Source: §4.1 · p. 4

4.2 Two-Tier Agentic Architecture

  • A top-level orchestrator owns database interaction and attempt history, while a tool-only NLToSQL generator returns one query from assembled context.The orchestrator invokes KGSchemaGetter, NodeSchemaGetter, and NLToSQL; the generator neither calls tools nor sees execution results. Frontier models handle orchestration and synthesis, while a faster, cheaper model handles constrained verification and feedback tasks.Source: §4.2 · p. 4, Figure 1 · p. 5

4.3 LLM-Driven Iterative Schema Discovery

  • LLM-driven schema discovery browses a precomputed knowledge graph, retrieves compressed nested schemas, and expands table context when execution exposes missing coverage.Offline preparation collapses partitions, scores key columns, and normalizes types; runtime catalog selection uses the question, key-column-enriched catalog, and cheatsheet without an embedding ranker. Parallel per-table retrieval returns Schemonic field names, types, enums, descriptions, and nested sub-fields; missing tables, columns, or aliases trigger additional retrieval.Source: §4.3 · pp. 4–5

4.4 Pre-Execution Query Plan Verifier

  • An LLM Query Plan Verifier checks join keys, aggregation grain, and deduplication placement before execution, routing structural flaws directly to regeneration without database access.Checks address Cartesian products and fan-out from joins or flattened arrays, reducing database compute and plausible-looking incorrect results.Source: §4.4 · p. 5

4.5 Dynamic Error Taxonomy and History-Aware Feedback

  • 15+ error types classify execution failures by phase and corrective action, routing schema-coverage failures to context expansion and other failures to targeted repair.Repair directives and the full failed-attempt history enter subsequent generator calls to prevent repeated mistakes. Error definitions and directives are configuration-driven and extensible without code changes.Source: §4.5 · pp. 5–6

4.6 Deterministic Checker Chain

  • Deterministic blockers reject semantic hazards, while nonfailing rewriters normalize identifier casing and translate cross-dialect functions; cheap static checks precede database execution.Blocker failures return classified errors to the repair loop; rewriters silently pass transformed SQL onward.Source: §4.6 · p. 6

4.7 Test-Time Learning

  • The Dynamic Cheatsheet framework of Suzgun et al. (2026) converts classified execution errors into transferable SQL rules, persisting them across evaluation and injecting top rules before generation.Rules can encode timestamp scaling requirements or the appropriate array-flattening operation.Source: §4.7 · p. 6

4.8 Semantic Validation

  • After successful execution, a lenient LLM validator compares the question, SQL, and result preview, rejecting clear semantic contradictions and triggering another generation attempt.It accepts directionally plausible outputs, including empty results under restrictive filters; rejection explanations are appended to attempt history.Source: §4.8 · p. 6

5 Experiments

The system achieves 91.7% Answer Correctness on DevRev-900 at $0.57 per correctly answered query, with little variation across difficulty buckets. Ablations identify important contributions from metadata and structured repair, while SDS-guided hints improve baseline accuracy without architectural changes.

5.1 Experimental Setup

  • All four systems use gpt-5.2-2025-12-11 without fine-tuning; APEX-SQL uses one vote, while FlexSQL retains four candidate plans.Baselines are APEX-SQL (Cao et al., 2026a), FlexSQL (Pham et al., 2026), and majority-vote ReFoRCE (Deng et al., 2025a); the proposed system routes verification and feedback to gpt-4.1-mini to reduce cost. Answer Correctness requires passing 10 deterministic checking stages, successful database execution, and GPT-5.2 semantic validation; secondary Mean Score normalizes a 50-point rubric, while Spider 2.0-Snow uses EX under the official script from Lei et al. (2025).Source: §5.1 · pp. 6–7

5.3 Difficulty-Stratified Analysis

  • 89.2%, 94.4%, and 91.0% Answer Correctness across Easy, Medium, and Hard buckets contrast with monotonically declining baseline scores.SDS buckets are Easy (≤12; 139 queries), Medium (13–18; 250), and Hard (>18; 511); system mean scores are 80.1%, 80.9%, and 78.1%. Its correctness advantage over FlexSQL grows from 44.6 to 54.8 to 57.1 percentage points.Source: Figure 2 · p. 7, §5.3 · p. 7

5.4 SDS as an Actionable Improvement Signal

  • 63.2% AC for ReFoRCE and 41.4% for APEX-SQL follow SDS-guided hints, improving their original 29.1% and 27.2% scores, respectively.Hints achieve 48.1% AC on ReFoRCE’s 638 originally failing queries and 19.5% on APEX-SQL’s 655 originally failing queries. The intervention inspects D1–D7 scores and injects dimension-specific natural-language guidance into schema context.Source: §5.4 · p. 7

5.2 Main Results

  • 91.7% Answer Correctness on DevRev-900 exceeds the next-best baseline, FlexSQL at 37.1%, by 54.6 percentage points.The system correctly answers 825/900 queries. Baselines cluster between 27–37%; the authors attribute their limitations to nested ARRAY[STRUCT] fields and typed link traversal rather than their different generation strategies.Source: §5.2 · p. 7, Table 2 · p. 8
  • $0.57 per correctly answered query makes the system 8× cheaper than APEX-SQL, 27× cheaper than FlexSQL, and 1.6× cheaper than ReFoRCE.Corresponding costs are $4.60 for APEX-SQL, $15.35 for FlexSQL, and $0.93 for ReFoRCE. Although ReFoRCE costs only $0.27 per query, its 29.1% correctness raises its cost per useful answer.Source: §5.2 · p. 7, Table 2 · p. 8

5.4 SDS as an Actionable Improvement Signal

  • +39.6 pp for Ranking/Top-N and +35.5 pp for Date/Time are ReFoRCE’s largest category gains from SDS-guided hints.Improvements span all six discriminative categories; these two categories map to D4 (multi-stage logic) and D6 (temporal precision). High-D4 and high-D2 (derived metrics) failures respond most strongly, supporting SDS as predictive and actionable without architectural changes.Source: §5.4 · p. 7, Table 3 · p. 8

5.5 Ablations

  • −15.0 pp overall accuracy and −64.4 pp on high-D7 queries make removing metadata-enriched retrieval the largest ablation effect.This ablation retains column names and DDL types but removes sub-field schemas, enum vocabularies, sample values, and join hints. Disabling iterative schema expansion reduces accuracy by 8.5 pp overall and 11.4 pp on Relationship/Network queries, highlighting its role in polymorphic link traversal.Source: §5.5 · pp. 7–8, Figure 3 · p. 8
  • 13.2 pp lower accuracy follows replacing structured error feedback with generic retries; removing deterministic checkers reduces accuracy by 5.9 pp.Generic retries omit classification, targeted directives, and history preservation, increasing mean repair rounds from 1.4 to 3.2. Removing pre-execution checkers increases mean database executions per query from 1.2 to 2.8, showing their cost and latency benefits.Source: Figure 3 · p. 8, §5.5 · p. 8

6 Conclusion · B System Design Details · B.1 Offline Knowledge Graph Construction

The paper pairs a benchmark for nested enterprise schemas with a cost-aware single-generation NL-to-SQL architecture. Offline knowledge-graph construction uses partition collapsing, key-column scoring, and type normalization.

6 Conclusion

  • The DevRev benchmark and cost-aware single-generation architecture provide complementary contributions addressing nested-type traversal and polymorphic link-graph reasoning in enterprise NL-to-SQL.The architecture uses iterative schema selection, sub-field-aware metadata retrieval, and a structured history-aware error taxonomy.Source: §6 · p. 8
  • The authors identify tournament-style multi-candidate generation as the most direct extension and expect design principles to transfer to dialects and enterprise platforms with similar schema properties.Source: §6 · p. 8

Limitations

  • Execution-API dependence prevents offline operation, while Snowflake-only evaluation leaves generalization to other SQL dialects unvalidated.Evaluation covers two benchmarks. DevRev uses LLM-assisted generation and execution verification against the production database, a different verification standard from fully human-curated benchmarks.Source: Limitations · p. 8
  • Cost comparisons with multi-candidate systems depend on disclosed candidate and execution-call counts; comparisons involving unreported costs are inferential.For systems without publicly reported costs, candidate count N is treated as a lower bound on inference cost.Source: Limitations · pp. 8–9

B.1 Offline Knowledge Graph Construction

  • Millions of physical tables become thousands of logical catalog entries by collapsing date-partitioned tables and retaining partition patterns as metadata.The knowledge graph is built once per deployment from source schema documentation.Source: §B.1 · p. 10
  • Key-column scoring uses uniqueness, naming heuristics, and foreign-key references to surface likely keys in the catalog, aiding table selection without full schema retrieval.Source: §B.1 · p. 10
  • Type normalization maps source-specific type strings to canonical ARRAY[STRUCT], STRUCT, and primitive types so downstream consumers reason uniformly about nested structures.Source: §B.1 · p. 10

B.2 Error Taxonomy Classes · B.3 Deterministic Checker Chain

The orchestrator maps SQL failures to phase-specific error classes and corrective actions. An ordered deterministic checker chain rejects or rewrites SQL before final database execution.

B.2 Error Taxonomy Classes

  • 15+ error types classify SQL failures by failure phase, with each class mapped to a corrective action.The taxonomy distinguishes schema-coverage, nested-type, identifier, and semantic errors.Source: §B.2 · p. 10
  • Schema-coverage errors—table_not_found, column_not_found, and unresolved_alias—trigger Stage 4 schema expansion.These failures expand schema coverage rather than triggering construct-specific or semantic repair.Source: §B.2 · p. 10
  • Nested-type errors trigger repair with construct-specific directives for forbidden_lateral, unnest_single_list, and nested_conversion_error.These cover incorrect LATERAL placement, flattening a singleton array, and type coercion over a sub-field, respectively.Source: §B.2 · p. 10

B.3 Deterministic Checker Chain

  • Ten ordered checker stages use blockers to reject SQL and rewriters to silently transform it, with database execution last.Blockers return classified errors to the orchestrator’s repair loop; statically detectable failures are handled before incurring database cost.Source: Table 4 · p. 10, §B.3 · p. 10

B.2 Error Taxonomy Classes

  • Identifier errors trigger alias-rewriting repair for reserved_word_alias and ambiguous_flatten_alias.These detect aliases colliding with Snowflake reserved words or conflicting aliases across multiple flattens.Source: §B.2 · p. 10
  • Semantic errors—empty_result, wrong_aggregation_grain, and missing_filter—trigger semantic repair with the question re-presented alongside the failed SQL.Source: §B.2 · p. 10

C Dataset Generation Pipeline: Full Details … D Additional Experimental Results

The dataset pipeline combines focused sub-schemas, formula-guided planning and generation, and persona rewriting that preserves analytical meaning. Separate rubrics assess natural-language query depth and generated SQL.

C Dataset Generation Pipeline: Full Details; C.1 Stage 0: Sub-Schema Generation

  • Across 14 table combinations, 150 sub-schemas provide focused Stage 1 inputs, selecting 2–5 analytically useful non-connection columns per table while retaining all primary-key and foreign-key columns.The generator receives full MSCHEMA source recording each field’s type, description, enum vocabulary, and sub-field schema for ARRAY[STRUCT] columns, then outputs a JSON sub-schema.Source: §C.1 · p. 11

C.2 The Formula Book

  • 25 structural formulas come from the highest-SDS Spider 2.0-Snow queries, all scoring at least 27 on the 0–35 scale.Human-readable entries combine source queries, slot-variable macro skeletons, and micro-pattern phrases annotated with their SDS dimensions and required SQL patterns. Machine-readable entries additionally encode column-type requirements and compatibility tags used during retrieval.Source: §C.2 · p. 11

C.3 Stage 1: Six-Slot Query Planning

  • Stage 1 uses one LLM call per sub-schema to plan six queries with explicit analytical intents, column selections, formula requirements, and category-coverage constraints.Slots span 15–175 words and carry D1–D7 targets on a 0–2 planning scale, distinct from the 0–5 scoring scale. Plans are regenerated unless Aggregation covers all six slots, Relationship/Network at least five, Classification/Grouping four, Date/Time three, and Ranking/Top-N, Statistical, and Comparison at least two each.Source: §C.3 · pp. 11–12

C.4 Stage 2: Formula Retrieval (Programmatic)

  • Stage 2 retrieves formulas programmatically without LLM calls, combining category Jaccard similarity with structural-pattern and available-column-type compatibility.It retains up to five formulas per slot, prohibits formula reuse across slots within a sub-schema, and strips SDS scores from retrieved examples to prevent generator overfitting.Source: §C.4 · p. 12

C.5 Stage 3: Per-Slot Query Generation; C.9 Annotated Pipeline Examples

  • Stage 3 generates each query with one LLM call, using its plan, sub-schema, and a cross-schema formula example for structural guidance without DevRev-vocabulary leakage.7.6% of generated queries are discarded for empty execution results; outputs retain column roles, a 2–5-sentence blueprint, length-bounded query text, and provenance. The annotated account_enhancement_ticket_1, Slot 6 example traces a formula macro-pattern through all four stages to its final persona rewrite.Source: §C.5 · p. 12, §C.9 · p. 13

C.6 Stage 4: Persona Rewriting

  • 100% of persona-rewritten queries pass their SDS slot threshold, while 76.9% achieve exact column-plan match in the automated audit.One LLM call rewrites six queries into distinct business-role voices, each used once per sub-schema, banning categorical values except analytically central single-value filters. A mandatory 15-item checklist protects dates, temporal grain, filters, thresholds, aggregation grain, denominators, and ranking; validity requires the original query to remain a natural interpretation of the rewrite.Source: §C.6 · pp. 12–13

C.7 SDS Scoring Rubric (Full Specification)

  • 35 points is the SDS maximum: an LLM judge scores seven dimensions from natural-language query text alone, without SQL or schema.Calibration anchors place Easy near 7, Medium near 15–18, and Hard near 29. Scores of at least 3 require specific textual justification; the rubric rejects inflation from length or jargon and requires downward correction against the Hard anchor when warranted.Source: §C.7 · p. 13

C.8 LLM Judge Rubric

  • 50 points across four criteria define SQL evaluation, with a mandatory clause-mapped checklist and a J1 hard cap for entirely unaddressed requirements.The judge receives the question, generated SQL, ground-truth DDL, and first five result rows. Before scoring, it checks entity scope, filters, grouping, metrics, ranking, output columns, multi-stage structure, and comparison direction.Source: §C.8 · p. 13

D.1 Results on DevRev: Detailed Breakdown · D.2 SDS-Stratified Per-Bucket Breakdown · D.3 Query Complexity Categories

The system achieves 91.7% Answer Correctness on DevRev-900, while its mean judge score remains within a 2.8 percentage-point range across SDS buckets. Queries receive analytical and business-domain tags, with minimum analytical coverage enforced during generation.

D.1 Results on DevRev: Detailed Breakdown

  • Failures on maximum-score SDS dimensions implicate computed-value reasoning (D2), date-function handling (D6), and nested-type machinery, including LATERAL FLATTEN (D7).Table 9 stratifies accuracy for queries scoring 5 on each SDS dimension.Source: §D.1 · p. 13, Table 9 · p. 15

D.2 SDS-Stratified Per-Bucket Breakdown

  • 2.8 percentage points bound the system’s mean judge score range across three SDS buckets, while all three baselines decline monotonically as SDS increases.Table 10 reports this comparison across all four systems.Source: §D.2 · p. 13

D.3 Query Complexity Categories

  • Analytical categories include Aggregation, Statistical, Comparison, and Classification/Grouping, covering summaries, distributional statistics, contrasts, and categorical or rule-based partitions.Comparison requires conditional aggregation or a self-join and is a primary source of multi-stage logic; Classification/Grouping tests grouping on the correct dimension, including enum-constrained strings.Source: §D.3 · p. 14
  • Ranking/Top N, Date/Time, Relationship/Network, and Geographic categories test ordered selection, temporal functions, multi-hop graph traversal, and location-based filtering or grouping.Ranking requires ORDER BY ... LIMIT or a window function; Geographic applies only when the sub-schema contains a geographic column.Source: §D.3 · p. 14

D.1 Results on DevRev: Detailed Breakdown

  • 91.7% Answer Correctness (825/900) is achieved on DevRev-900, and 91.5% Answer Correctness (745/814) on the high-SDS subset (SDS ≥11).The high-SDS subset comprises 90.4% of the corpus.Source: §D.1 · p. 13, Table 8 · p. 15

D.3 Query Complexity Categories

  • Business-domain categories emerge from selected table combinations rather than enforced generation constraints, supporting downstream per-domain failure-rate analysis.Examples include Ticket/Support, Account/Customer, Opportunity/Sales, Product/Feature, Meeting/Engagement, and Dashboard.Source: §D.3 · p. 15
  • Six generation slots per sub-schema enforce minimum analytical-category coverage to prevent the corpus from collapsing to a single query type.Aggregation appears in all six; Relationship/Network in at least five; Classification/Grouping in at least four; Date/Time in at least three; Ranking/Top-N, Statistical, and Comparison each in at least two.Source: §D.3 · p. 15

D.4 Dataset Statistics · SDS Bucket Ours APEX-SQL FlexSQL ReFoRCE

DevRev contains 900 execution-verified queries across 14 schema combinations and exhibits analytical depth comparable to Spider 2.0-Snow, but with a different difficulty profile.

D.4 Dataset Statistics

  • 900 execution-verified natural-language queries span 14 schema combinations in the DevRev corpus.Table 12 compares corpus-level statistics with Spider 2.0-Snow.Source: §D.4 · p. 15

SDS Bucket Ours APEX-SQL FlexSQL ReFoRCE

  • 89.2% (Easy), 94.4% (Medium), and 91.0% (Hard) are our system’s Answer Correctness figures across SDS buckets.These are Answer Correctness figures, distinct from Table 10’s mean judge scores expressed as percentages of a 50-point maximum.Source: Table 10
  • Queries receive multiple complexity-category labels, with percentages calculated over all 900 queries; aggregation is universal by design.Table 11 reports the fraction of queries containing each category, rather than mutually exclusive category shares.Source: Table 11 · p. 16
  • 19.15 versus 20.20 mean SDS indicates comparable analytical depth for DevRev and Spider 2.0-Snow, respectively, despite different difficulty profiles.Both corpora were scored using the same 0–5-per-dimension rubric.Source: Table 12 · p. 16, SDS Bucket Ours APEX-SQL FlexSQL ReFoRCE · p. 16
  • 3.67 versus 2.57 for SQL-construct diversity (D7), and 3.12 versus 2.72 for multi-stage pipeline logic (D4), distinguish Spider 2.0-Snow from DevRev.Spider 2.0-Snow concentrates difficulty in these dimensions, reflecting cross-domain schema variety and advanced SQL patterns.Source: SDS Bucket Ours APEX-SQL FlexSQL ReFoRCE · p. 16
  • 2.92 versus 2.65 for derived-metric construction (D2) illustrates DevRev’s difficulty profile relative to Spider 2.0-Snow.DevRev also scores higher on output-specification precision (D5: 3.45 versus 2.88) and temporal reasoning (D6: 1.51 versus 1.33). These dimensions reflect CRM requirements such as percentage-change comparisons and rolling-window time filters.Source: SDS Bucket Ours APEX-SQL FlexSQL ReFoRCE · p. 16
  • 0.039 self-BLEU indicates maintained lexical diversity despite all DevRev queries sharing domain vocabulary.Self-BLEU measures n-gram redundancy; lower values indicate greater diversity.Source: SDS Bucket Ours APEX-SQL FlexSQL ReFoRCE · p. 16, Table 12 · p. 16

D.5 Results on Spider 2.0-Snow

The system achieves 85.8% Execution Accuracy on Spider 2.0-Snow’s 120-query public gold-answer subset using one generation pass per query. This result is indicative rather than directly comparable to baseline scores on the full 547-query leaderboard.

D.5 Results on Spider 2.0-Snow

  • 85.8% Execution Accuracy—103 correct answers out of 120—was achieved on Spider 2.0-Snow’s public gold-answer subset.Only 120 of 547 queries have publicly available ground-truth SQL and expected results (Lei et al., 2025). The subset score is not directly comparable to full-leaderboard figures and was not submitted to the ranked leaderboard.Source: Table 13 · p. 16, §D.5 · p. 16
  • 63–73% EX on Spider 2.0-Snow’s full leaderboard contrasts with 27–37% AC on DevRev for the three baselines.The baselines are APEX-SQL, FlexSQL, and ReFoRCE, each represented by its best leaderboard submission. The paper attributes their DevRev performance gap to enterprise nested-schema structure rather than general system weakness.Source: §D.5 · p. 16, Table 13 · p. 16
  • The gold-subset result suggests competitiveness in flat-table, large-scale schema linking while using one generation pass per query.The authors qualify this conclusion because their evaluation covers only the publicly released subset.Source: §D.5 · pp. 16–17

D.6 Cost Comparison

The system uses fewer LLM calls than APEX-SQL and FlexSQL while matching ReFoRCE’s database-execution footprint at higher answer correctness. Its single metadata-guided schema-selection pass avoids repeated exploratory probing.

D.6 Cost Comparison

  • 8–23× fewer LLM calls than APEX-SQL and FlexSQL accompany 3× higher Answer Correctness than ReFoRCE, while matching ReFoRCE’s median 4 database executions.Cost counts are medians from the same stratified 50-query sample, measured using API logs and execution traces. Answer Correctness covers the full 900-query DevRev benchmark.Source: Table 14 · p. 17, §D.6 · p. 17
  • 2 LLM calls per query (median; maximum 3) reflect one SQL-generation call plus repairs triggered by the pre-execution checker or execution failures.A single schema-selection pass guided by the metadata-enriched retriever avoids the multi-candidate baselines’ independent exploratory schema probes.Source: §D.6 · p. 17
  • 46 median LLM calls and 20 median database executions characterize FlexSQL, which generates 4 candidate plans.Each plan independently executes subqueries per involved table to ground column values before SQL synthesis.Source: §D.6 · p. 17
  • 16 LLM calls per query (mean and median) and 18 median database executions characterize APEX-SQL’s hypothesis-verification loop.Each LLM call is paired with a database execution to verify or refute a schema hypothesis.Source: §D.6 · p. 17
  • 2× more LLM calls than our system are used by ReFoRCE, despite its matching database-execution footprint.Its self-refinement loop runs for up to 5 iterations, each issuing one LLM call and one database execution.Source: §D.6 · p. 17

Related papers

Loading 2609.04641v1…