PaperlayerView exact revision on arXiv

Source-linked AI summary

ESQ-Bench: A Multi-Tier Enterprise Oracle Benchmark for Evaluating NL2SQL Dialect Generalization and Silent Semantic Divergence

Sanjay Mishra, Divya Chukkapalli, Ganesh R. Naik

arXiv:2608.23569v1cs.AI

TL;DR

Enterprise NL2SQL benchmarks underrepresent Oracle dialects and enterprise schema complexity, limiting evidence about production-oriented correctness. ESQ-Bench addresses this with an Oracle-first, tiered benchmark and silent-divergence evaluation, finding monotonic GPT-4o execution-match degradation across tiers and substantial semantic risk among execution-passing queries. The authors also identify evaluation-scope boundaries requiring further validation.

  • Problem

    Existing NL2SQL benchmarks use simplified schemas and open SQL dialects, leaving enterprise Oracle complexity and silent semantic divergence insufficiently evaluated.

  • Method

    ESQ-Bench combines six populated enterprise-representative schemas, 550 gold-validated question-query pairs, four metrics, and Oracle-first evaluation across three complexity tiers.

  • Results

    79.8% →60.3% →57.2% EX shows monotonic GPT-4o schema-linked degradation across tiers, while 92–99% SDop among EX-passing queries indicates frequent semantic divergence.

  • Takeaways & Limitations

    Enterprise Oracle NL2SQL evaluation should examine schema-complexity scaling and semantic correctness beyond execution match.

  • Takeaways & Limitations

    Spider/BIRD replication on the harness remains future work to determine whether schema-linked tier degradation is benchmark-driven rather than an implementation artifact.

Abstract

from arXiv · show

State-of-the-art Natural Language to SQL (NL2SQL) models report execution accuracy exceeding 89 percent on established benchmarks such as Spider and BIRD. However, these benchmarks rely on simplified academic schemas and open-source SQL dialects that do not reflect the complexity of enterprise database environments. We introduce ESQ-Bench, an Oracle-first NL2SQL benchmark with systematic complexity tiers and silent-divergence evaluation across three enterprise schema complexity tiers. We constructed and released six populated schemas (465 tables, 164,682 rows, zero empty tables) with identical seed data on Oracle, PostgreSQL, MySQL, and SQL Server, a four-metric evaluation harness (EM, EX, SR, SD), and 550 gold-validated question-query pairs (Tier-1: 95; Tier-2: 228; Tier-3: 227). Schema-linked prompting with GPT-4o shows monotonic execution-match degradation across tiers: 79.8, 60.3, and 57.2 percent EX on executed queries (June 2026), versus 75.6, 80.4, and 95.8 percent on an earlier 142-question pilot slice. EM stays below 7 percent tier-wide; operational silent-divergence reaches 73 to 99 percent among EX-passing queries. Failure analysis shows wrong-result semantics dominate at higher tiers. Claude Sonnet 4.6 with schema-linked prompts reaches 87.4, 74.9, and 68.7 percent EX (executed queries), exceeding GPT-4o schema-linked on every tier. GPT-4o zero-shot EX on executed queries (78.7, 73.5, and 77.8 percent) inverts schema-linked at Tiers 2 to 3 due to lower execution rates and survivor bias in the zero-shot versus schema-linked analysis. Local Llama 3.2 schema-linked reaches only 13.3 percent bank-wide EX (73 out of 550), underscoring the gap between closed API models and open-weight baselines on enterprise Oracle schemas.

1 Introduction

ESQ-Bench addresses the gap between optimistic NL2SQL benchmark results and enterprise deployment by testing Oracle-oriented schema complexity and silent semantic divergence. It contributes a populated, multi-dialect benchmark, four-metric evaluation, and complete gold-validated question bank with full-model baselines.

  • Motivation: 89%+ reported execution accuracy on established benchmarks contrasts with a poorly understood gap between benchmark and enterprise production accuracy.The paper links this gap to enterprise schema complexity and SQL dialect characteristics absent from published benchmarks.
  • Motivation: Enterprise Oracle schemas routinely contain 150–500 tables, partial foreign-key enforcement, ambiguous names, legacy abbreviations, and denormalized structures.These characteristics differ sharply from Spider’s average of 5.1 tables, complete foreign-key documentation, and clean naming conventions.
  • Motivation: Oracle-specific syntax and semantics, including FETCH FIRST, CONNECT BY, MINUS, empty-string-as-NULL coercion, and NULLS LAST ordering, can produce failing or silently wrong queries.The paper frames dialect mismatch as a distinct source of enterprise evaluation risk.
  • Motivation: Execution Match cannot detect semantically incorrect queries that happen to return the gold rows on one test instance, motivating silent-divergence measurement.The cited metric critique highlights Oracle-specific NULL handling, implicit coercion, and boundary-condition differences.
  • Contributions: ESQ-Bench provides six enterprise-representative schemas with 465 tables and 164,682 seeded rows across three tiers, using identical data on Oracle, PostgreSQL, MySQL, and SQL Server.The benchmark is Oracle-first while supporting cross-dialect infrastructure.
  • Contributions: 550 gold-validated question-query pairs span Tiers 1–3, with 95, 228, and 227 pairs respectively and critical-condition ties to seeded data.The conditions include NULL traps, status-domain separation, CONNECT BY hierarchies, entity-overlap joins, and ambiguity-resolution traps.
  • Results: 79.8% / 60.3% / 57.2% tier execution match is reported for full-bank GPT-4o schema-linked evaluation, with EM below 7% and operational silent divergence at 73–99%.The evaluation also reports monotonic complexity degradation and an F1–F4 failure taxonomy.
  • Contributions: The public release includes schemas, seed scripts, all 550 questions, the evaluation harness, failure taxonomy, and GPT-4o and Claude Sonnet 4.6 baselines.This supports reproduction of the benchmark and its complete-bank evaluations.

2 Background

The paper defines NL2SQL as generating a query whose result satisfies a question’s intent, then explains why Oracle enterprise evaluation requires dialect-aware semantics beyond row-set execution matching. Oracle’s distinct syntax, hierarchy support, functions, and NULL behavior create both explicit failures and silent divergences.

  • Task Definition: NL2SQL generates SQL q over schema S so execution against database instance D returns result set R(q, D) satisfying the natural-language question’s intent.The task definition formalizes schemas as tables with columns, data types, and constraints.
  • Task Definition: Matching the gold result set on one instance is necessary but insufficient because queries may diverge on other data distributions.The paper emphasizes Oracle-specific NULL semantics and implicit type coercion as sources of exposed or hidden divergence.
  • Oracle Dialect: Oracle includes enterprise-relevant syntax absent from existing NL2SQL benchmarks, including FETCH FIRST row limiting and CONNECT BY hierarchical traversal.Models trained on SQLite or PostgreSQL may generate dialect-incompatible or semantically different queries.
  • Oracle Dialect: Oracle uses MINUS rather than EXCEPT, so models generating EXCEPT can receive ORA-00933 errors.This is an explicit dialect failure rather than a silent semantic divergence.
  • Oracle Semantics: Oracle treats empty strings as NULL and defaults ascending NULL ordering to NULLS LAST, creating silent divergence when NULL-bearing data exposes these differences.The contrasting defaults are invisible in test data without NULL values.
  • Oracle Semantics: Oracle-specific functions such as DECODE, NVL, NVL2, LISTAGG, SYS_CONNECT_BY_PATH, and RATIO_TO_REPORT lack direct equivalents in other dialects.Approximate substitutions from open-dialect training data may produce subtly different results.

3 Related Work

Prior NL2SQL benchmarks expanded scale, schema diversity, dialogue, robustness, and enterprise relevance, but generally retain simpler schemas or non-Oracle execution targets. ESQ-Bench positions itself as complementing these efforts through Oracle-specific constructs, complexity tiers, and silent-divergence evaluation.

  • Earlier Benchmarks: 80,654 WikiSQL examples enabled large-scale evaluation but were limited to single-table queries.That limitation precluded multi-join and subquery evaluation.
  • Earlier Benchmarks: Spider introduced 10,181 examples across 200 databases with cross-domain generalization and complex SQL, establishing execution match as the standard metric.Its schemas average 5.1 tables, enforce and document foreign keys, use clean naming, and target SQLite.
  • Robustness and Interaction: SParC and CoSQL added multi-turn and dialogue-based interaction while retaining Spider’s schema simplicity and SQLite dialect.Their extensions target interaction structure rather than enterprise Oracle complexity.
  • Robustness and Interaction: Spider-Syn tested synonym-based lexical robustness, whereas Dr. Spider introduced 17 systematic perturbation categories without addressing enterprise schemas or Oracle semantics.Dr. Spider is identified as the closest motivating robustness study but remains constrained to Spider and SQLite.
  • Real-Database Evaluation: 272 KaggleDBQA questions evaluated eight non-curated databases, but the scale was insufficient for statistically reliable category-level claims and none targeted Oracle.This provided real-schema coverage without the enterprise Oracle focus of ESQ-Bench.
  • Real-Database Evaluation: BIRD expanded to 12,751 questions and larger schemas averaging 7.3 tables, yet execution remained on SQLite or PostgreSQL and below enterprise complexity.BIRD is described as the current standard for challenging NL2SQL evaluation.
  • Enterprise-Relevance: Spider 2.0 targets enterprise-grade warehousing and analytical tasks but lacks Oracle-specific constructs, systematic schema-complexity tiers, and a silent-divergence metric.ESQ-Bench complements it by isolating Oracle dialect and silent-divergence effects on seeded enterprise-representative schemas.
  • Comparison: Table 1 compares benchmark table counts, enterprise characteristics, real databases, Oracle dialect coverage, and silent-divergence metrics.The paper states that prior benchmarks have empty Oracle-dialect and silent-divergence columns.

4 ESQ-Bench Design

ESQ-Bench is designed as a diagnostic, Oracle-first benchmark whose three complexity tiers make degradation measurable and whose semantic annotations support evaluation beyond execution match. Its reproducible schemas, seeded critical conditions, quantified complexity features, and blind-verified question protocol target enterprise failure modes.

  • Design Principles: ESQ-Bench is a diagnostic instrument governed by tiered complexity, Oracle-first design, metric completeness, and full reproducibility.These principles define the benchmark’s scope and evaluation philosophy.
  • Design Principles: Three complexity tiers make degradation curves and failure thresholds measurable rather than treating schema complexity as binary.The design explicitly supports analysis across increasing schema difficulty.
  • Design Principles: Every question, schema, and gold query targets Oracle Database through a ground-up design for Oracle semantics, syntax, and enterprise conventions.This is not a dialect translation of an existing benchmark.
  • Design Principles: Semantic intent annotations and distractor queries extend evaluation beyond execution match through the Silent Divergence Rate metric.The benchmark is intended to distinguish executable but semantically wrong answers.
  • Infrastructure: A single manifest drives dialect-aware DDL generation and deterministic seeding, with fixups injecting NULL fractions, status-domain separation, and hierarchy depths.The current release reports all six schemas fully defined, populated, and verified.
  • Infrastructure: 550/550 gold queries execute on Oracle, and the zero-empty-tables invariant prevents execution-match evaluation from passing vacuously on empty relations.Scalar expectation checks validate seeded critical conditions.
  • Complexity Features: AI, FCR, OCD, and LNR quantify ambiguity, foreign-key coverage, Oracle-construct density, and legacy naming across tiers.These four features operationalize schema complexity profiles.
  • Tier Profiles: Tier 1 replicates Spider’s upper-quartile complexity with enforced foreign keys and standard naming, while Tier 2 introduces partial enforcement, mixed naming, and ambiguity.T1-A includes ten tables and basic Oracle constructs; T2-A has 48 tables, status-domain ambiguity, NULL conditions, and analytic functions.

5 Evaluation Metrics

ESQ-Bench evaluates NL2SQL with complementary metrics that distinguish exact query form, result-set agreement, partial row recovery, and semantic divergence. Its framework explicitly tests whether execution success can overstate correctness through distractor queries and supplementary data instances.

  • Execution Match (EX): EX compares generated and gold result sets, using set-based equality except when question semantics require ordering.EX is the primary comparability measure, but cannot detect silent semantic divergence.
  • Semantic Recall (SR): SR measures the fraction of intended rows returned and captures over-filtering failures that EX scores as zero.Mean SR over EX-failing questions quantifies how close incorrect answers are.
  • Silent Divergence Rate (SD): Silent divergence occurs when an EX-passing query matches a distractor result or violates an explicit intent constraint on a supplementary instance.Intent constraints include NULL handling, boundary conditions, and ordering requirements.
  • Silent Divergence Rate (SD): SD is benchmark-level, applies only to EX-passing questions, and a non-zero value indicates that execution match over-reports accuracy.The operational flag is computed without distractor matching, while formal SD targets questions with authored distractors and supplementary instances.
  • Failure Mode Taxonomy: The failure taxonomy classifies EX failures as syntactic invalidity, valid-standard but invalid-Oracle SQL, wrong-result execution, or non-idiomatic queries.The taxonomy separates Oracle-specific rejection from queries that execute but return results differing from gold.

6 Experiments

Experiments evaluate GPT-4o and other models on the full Oracle-first ESQ-Bench using schema-linked and zero-shot prompting. GPT-4o execution match declines with tier complexity, while silent divergence and wrong-result failures expose limitations that execution match alone misses.

  • Tier-1 Findings: 100% EX for simple selects, aggregations, and set operations contrasts with 44.4% conditional-logic and 66.7% multi-join EX in Tier 1.Ordering reaches 50.0% EX on the expanded bank.
  • Tier-1 Findings: 6.3% EM versus 79.8% EX at Tier 1 shows that execution match substantially exceeds exact SQL agreement.Among 75 EX-passing queries, 69 trigger operational silent divergence because EM equals zero.
  • Tier-2 Results: 60.3% Tier-2 EX includes 96.2% aggregation EX but 0% hierarchical EX on executed queries despite schema-linked hints.The hierarchical category involved four-table organization trees.
  • Tier-3 Results: 57.2% Tier-3 EX coexists with 86.7% hierarchical EX under explicit CONNECT BY hints, while ambiguity resolution, multi-join, and ordering remain weak.Those categories reach 0%, 23.5%, and 10.0% EX respectively.
  • Cross-Tier Results: 79.8%, 60.3%, and 57.2% EX mark GPT-4o’s monotonic decline across Tiers 1–3 on executed queries.The full bank contains 95, 228, and 227 questions respectively.
  • Failure Analysis: F3 wrong-result semantics account for 170 of 243 failures, and operational silent divergence reaches 92–99% among EX-passing queries across higher-tier runs.F1 invalid-identifier errors rise from 1 in Tier 1 to 42 across Tiers 2–3 combined.
  • Baselines and Prompt Conditions: Claude Sonnet 4.6 exceeds schema-linked GPT-4o on EXexec at every tier, while zero-shot GPT-4o appears stronger on executed Tiers 2–3 because of survivor bias.Schema linking improves execution rates but can expose harder multi-table and Oracle-construct errors.

7 Complexity Predictor Analysis

The analysis models execution-match failure using nine schema and question complexity features. Join depth, schema size, ambiguity-related naming, and Oracle construct density associate with lower EX, while several features are unidentifiable because they lack variance.

  • Regression Setup: 66.0% in-sample accuracy comes from a balanced logistic regression predicting binary EX across 550 questions using nine standardised features.The positive EX rate is 55.8%.
  • Predictors: β = −0.68 makes per-question join depth the strongest negative predictor of EX.Deeper joins correlate with execution-match failure.
  • Predictors: AI has only β = +0.06, indicating a weak positive association between ambiguity index and EX at this scale.This association runs counter to the stated deployability hypothesis.
  • Feature Limitations: FCR, DD, and NP coefficients are not identifiable because each feature has zero variance across the six v0.3 schemas.The schemas have uniform FK coverage, no column comments, and identical nullable-column ratios.
  • Silent-Divergence Probes: Four NULL-trap probes achieve 100% EX, versus 62.5% EX on the remaining 485 executed non-probe questions.Confirmed formal silent-divergence instances for Tier 2–3 distractors remain future work.
  • Deployment Rubric: The resulting rubric recommends measuring ambiguous columns, FK coverage, Oracle construct requirements, documentation, and average join depth before deployment.Adding comments to highly ambiguous tables is presented as an actionable prompt aid.

8 Discussion

ESQ-Bench shows that enterprise-like complexity reduces NL2SQL accuracy and that execution success alone does not establish semantic correctness. The discussion recommends richer validation and highlights scope, measurement, version, and Oracle-release boundaries.

  • Accuracy and validation: 79.8% to 57.2% EX across Tiers 1 to 3 shows enterprise-like complexity materially reduces GPT-4o accuracy.Tier-3 performance falls below published Spider/BIRD results for comparable models on SQLite.
  • Accuracy and validation: 92.0% operational SD among EX-passing Tier-1 queries shows execution success alone can conceal non-gold SQL paths.The paper recommends tracking EM, SR, formal SD where available, semantic intent, supplementary instances, and SQL paths.
  • Practitioner implications: Join depth and legacy naming ratio are the strongest regression predictors of EX failure, making documentation of ambiguous tables an actionable intervention.Documentation density could not be estimated in v0.3 because DD=0 across all six schemas.
  • Benchmark implications: Future benchmarks should combine enterprise-complexity tiers with at least one production commercial dialect and report SD alongside EX.The authors describe SD as qualitatively different from EX despite the non-trivial effort required to construct semantic intents and distractor queries.
  • Scope and limitations: The release contains 550 questions, but few-shot and DAIL-SQL baselines remain future work.Tier-level comparisons are supported by category-level statistics, including N = 10 for Tier-3 ambiguity resolution.
  • Scope and limitations: ESQ-Bench uses enterprise-like rather than real production schemas, and results are scoped to Oracle Database 21c with pinned model versions.Oracle 23ai constructs such as JSON relational duality and AI vector search are not included.
  • Scope and limitations: Formal SD for Tiers 2–3 is still being authored, while current high Tier-1 SDop mainly reflects EX ∧¬EM rather than confirmed distractor matches.SD-probe questions show 100% EX but expose silent NULL risk when models omit NVL; formal SD on seed data currently yields zero confirmed divergences.

9 Open Problems

The paper identifies open problems in Oracle-aware training, automated silent-divergence detection, complexity adaptation, multi-turn evaluation, and cross-dialect transfer. These questions target both model improvement and broader enterprise evaluation coverage.

  • Training and detection: Oracle-dialect fine-tuning remains untested at scale, including the achievable accuracy gain and minimum training-set size.The 550 question-query pairs provide a starting corpus for this investigation.
  • Training and detection: Automated SD detection could replace currently human-authored semantic intent descriptions through semantic comparison, equivalence checking, or constraint extraction.The stated goal is production monitoring that can identify silent divergence automatically.
  • Complexity adaptation: Schema-linked prompting partially mitigates complexity-driven degradation but does not close the Tier-3 gap, motivating curriculum, augmentation, and Oracle-specific pretraining studies.The open question concerns which intervention improves Tier-3 performance most efficiently.
  • Evaluation scope: ESQ-Bench is single-turn, leaving multi-turn enterprise accuracy and conversational safety-risk accumulation entirely unmeasured.The proposed evaluation target is iterative interaction on enterprise schemas.
  • Evaluation scope: Cross-dialect transfer studies could test whether Oracle-trained or evaluated models generalize to SQL Server T-SQL and DB2 SQL/PL without dialect-specific training.This would assess whether enterprise complexity generalizes across commercial databases.

10 Conclusion

ESQ-Bench contributes an Oracle-first, multi-tier benchmark with silent-divergence evaluation and reproducible resources. Its full-bank results show monotonic complexity degradation and a sharp divergence between execution and exact match, while the release supports practical validation and further research.

  • Contribution: ESQ-Bench introduces systematic complexity tiers and silent-divergence evaluation for Oracle-first NL2SQL assessment.The benchmark is designed around enterprise schema and dialect conditions.
  • Benchmark construction: 465 tables and 164,682 rows populate six schemas across four SQL dialects, with 550 gold-validated questions spanning three tiers.The schemas contain zero empty tables and use identical seeded data across dialects.
  • Findings: 79.8% / 60.3% / 57.2% EX across Tiers 1–3 confirms monotonic degradation, while EM remains below 7% and execution and exact match diverge sharply.These are full-bank GPT-4o schema-linked results.
  • Implications: 92% operational SD among EX-passing Tier-1 queries motivates semantic recall, failure-taxonomy review, and SQL-path auditing beyond execution success.The complete question bank and tools also support regression, multi-model baselines, and reproducible leaderboard extensions.
  • Resources: The released package includes schemas, seed scripts, questions, failure-taxonomy tools, and the evaluation harness.A community leaderboard is planned at the project repository.

A Historical Mini-Pilot (v0.1)

The historical v0.1 material preserves an early GPT-4o-mini Tier-1 pilot for longitudinal comparison. Its results should not be directly compared with full-bank GPT-4o results because the question set and model differ, and the 142-question slice used mixed models and pre-expansion banks.

  • Pilot design: GPT-4o-mini was evaluated on 46 Tier-1 questions in the early v0.1 mini-pilot.The pilot used schema-linked SAL in June 2026.
  • Comparability: The v0.1 numbers are preserved for longitudinal comparison rather than direct comparison with full-bank GPT-4o results.Both the question set and model differ between evaluations.
  • Pilot versus expanded bank: The 142-question cross-tier pilot used mixed models and pre-expansion banks, with full-bank GPT-4o EX of 75.6% / 80.4% / 95.8% on that slice.After expansion to 550 questions, full-bank GPT-4o EX was 79.8% / 60.3% / 57.2%.
Loading 2608.23569v1…