PaperlayerView exact revision on arXiv

Source-linked AI summary

DRL: A Deterministic Relational Middleware Layer for Transaction-Safe Enterprise NL2SQL Under Schema-Graph Scaling

Sanjay Mishra, Divya Chukkapalli, Ganesh R. Naik

arXiv:2608.26172v1cs.DBcs.AI

TL;DR

Enterprise NL2SQL loses reliability on large OLTP schema graphs because models face expanding, ambiguous structural search spaces and incomplete metadata. DRL inserts deterministic graph pruning, typed relational compilation, and plan-aware safeguards between natural-language front ends and SQL backends. On a 1,000-pair suite, the evaluated models clustered near 52–53% execution match, while the study also identifies evaluation-harness defects and synthetic-suite scope limits.

  • Problem

    Enterprise OLTP catalogs contain large schema graphs, incomplete foreign-key metadata, and namespace collisions that make full-catalog NL2SQL difficult to evaluate and deploy reliably.

  • Method

    DRL bounds a join-safe schema sub-graph before generation, compiles output through an ANSI-first RAST, and applies EXPLAIN, NULL, and silent-divergence safeguards.

  • Results

    The three evaluated models cluster within roughly one percentage point at 52–53% execution match under schema-linked prompts, with statistically indistinguishable Wilson intervals.

  • Takeaways & Limitations

    The results support treating enterprise NL2SQL as a middleware and measurement problem involving schema scaling, semantic verification, and plan-aware admission.

  • Takeaways & Limitations

    The suite uses six synthetic-but-realistic OLTP schemas rather than live production systems, so its structural rates are design parameters rather than population measurements.

Abstract

from arXiv · show

Deploying natural-language interfaces over enterprise OLTP catalogs fails at scale because semantic parsers collapse under schema-graph scaling, inflating context beyond stable LLM attention budgets. We present DRL (Deterministic Relational Middleware Layer), a safe pipeline interposing between front-ends and SQL backends. DRL comprises dynamic context pruning, relational AST typing, and transactional safeguard verification (EXPLAIN gating and NULL guards) to bound context and flag operational silent divergence (SDop). We evaluate DRL on PostgreSQL and MySQL, contributing (i) an OLTP schema-graph scaling model, (ii) a 1,000-pair Workload Verification Suite, (iii) baselines B0-B3, and (iv) an enterprise NL2SQL failure taxonomy. On PostgreSQL, schema-linked hints (B1) yield a 76% context reduction over naive full-catalog prompting (B0); DRL's dynamic router (B2) reaches a 92% reduction at pruning p95 = 0.58 ms and middleware p95 = 4.6 ms. GPT-4o, Claude Sonnet 4.5, and Gemini 2.5 Flash achieve 52.9%, 52.8%, and 52.1% execution match under a corrected evaluation harness; SDop flags 89-100% of false-positive EX-passing queries. GPT-4o failures are dominated by semantic/filter errors (254/471), while column hallucination is a minor factor (47/471). Crucially, a single regex defect in our evaluation post-processor silently suppressed accuracy and manufactured a false 4-10% cross-vendor gap that vanished when corrected, showing that benchmark code deserves the same scrutiny as the models it scores. DRL reframes enterprise NL2SQL as systems engineering - context bounding, verification, and plan-aware admission - not a leaderboard exercise.

1 Introduction

Enterprise NL2SQL accuracy degrades on large, structurally messy OLTP catalogs because models must implicitly resolve expanding schema search spaces. DRL addresses this with deterministic context bounding, typed compilation, safeguards, and reproducible evaluation infrastructure.

  • Motivation: 85–91% execution accuracy on small academic schemas falls to 52–53% on the paper’s 1,000-question enterprise suite.The enterprise schemas contain normalized tables, legacy abbreviations, duplicated column names, and incomplete catalog foreign keys.
  • Problem formulation: The systems-boundary problem arises because growing schema graphs lack deterministic mechanisms to bound context, compile output, and verify transactional semantics.The paper frames the degradation as a systems issue rather than primarily a model-capacity issue.
  • Motivation: DRL explicitly narrows a 168-table catalog to the two or three tables relevant to an intent before generation.This converts implicit in-context search into an auditable graph-traversal problem with a hard cardinality bound.
  • DRL architecture: DRL combines bounded sub-graph extraction, ANSI-first RAST compilation, and pre-execution verification for operational divergence and physical plan safety.The architecture includes implemented pruning, RAST validation, EXPLAIN gating, and NULL guards; some dialect-emission components remain specified rather than integrated.
  • Contributions: The paper contributes a schema-graph scaling formalization, a 1,000-pair verification suite, measured B0–B3 baselines, and a structural enterprise NL2SQL failure taxonomy.The suite and harnesses are released with PostgreSQL validation and a MySQL cross-check.
  • Evaluation: A shared evaluation-harness defect suppressed accuracy and manufactured an apparent cross-vendor gap, which disappeared after correction.The paper presents this as a general lesson that benchmark post-processing is part of the measurement instrument.

2 The OLTP Schema Graph Scaling Problem

The OLTP schema-graph problem combines rapidly growing join-path alternatives with incomplete and ambiguous catalog metadata. DRL bounds the routed sub-graph, reducing context growth while leaving semantic join selection as a residual limitation.

  • Structural sources of degradation: Tier 3 insurance schemas can exceed 4,200 column tokens before join paths, nullability, or active-flag conventions are included.This illustrates how full-catalog prompting expands context without necessarily adding disambiguating signal.
  • Formalization: Context scaling degradation is measured as the execution-accuracy gap between DRL-pruned contexts with |Vq| ≤5 and matched full-catalog prompts.The definition treats degradation as a marginal loss associated with schema-context size.
  • Structural sources of degradation: Enterprise catalogs contain application-maintained join paths, repeated identifiers, and surrogate-key projection drift that complicate semantic reconstruction.The cited schemas contain 23–41% application-maintained join paths, identifiers repeated in at least seven unrelated tables, and projection-related execution failures.
  • Formalization: O(d̄^h) join-path growth is reduced to a bound depending on the fixed routed sub-graph rather than the full catalog size.The theorem is a worst-case combinatorial statement for h-hop joins, not a correctness guarantee.
  • Residual limitation: The router does not guarantee correct join selection: lexical seed errors can propagate, and semantic/filter errors remain dominant after pruning.Its benefit is largest in multi-join and analytic-function categories where naive search grows fastest.
  • DRL response: DRL computes Gq before generation, compiles output to RAST, and rejects queries violating plan or NULL safeguards.These stages target context growth, dialect-independent structure, and operational divergence or safety risks.

3 Middleware Engine Architecture

DRL’s middleware pipeline combines graph-based context pruning, a relational AST representation, and pre-admission safeguards for SQL plans and NULL handling.

  • Architecture: The request path fully implements context pruning and safeguard verification, while the RAST emitter remains specified and only partially enforced.The full parser-and-emitter compiler and live dialect emission are not yet integrated into serving.
  • Dynamic Context-Pruning Router: The router lexically anchors questions to seed tables, computes one-hop foreign-key closure, and caps the selected schema at five tables.It falls back to domain-hint tables when lexical anchoring finds no seed, never reverting to the full catalog.
  • Dynamic Context-Pruning Router: Median selected-schema size is 3.2, 4.1, and 4.8 tables across Tiers 1, 2, and 3, versus 10–177 catalog tables; PostgreSQL pruning p95 is 0.76 ms.The corresponding p50 pruning overhead is 0.19 ms.
  • Relational AST Compiler: RAST provides an intermediate representation for mapping ANSI constructs to PostgreSQL and MySQL dialect syntax.Specified mappings include FETCH FIRST→LIMIT, COALESCE, and EXCEPT→NOT EXISTS.
  • Relational AST Compiler: Emitter rules enforce read-only statements, exact projections, NULL-safe aggregates, and dialect-aware value and ordering behavior.These rules target observed projection, aggregation, ordering, pagination, and string-encoded boolean failure modes.
  • Transactional Safeguard Verification: Before admission, the safeguard layer computes SDop, checks plans with EXPLAIN, and enforces COALESCE around aggregates on nullable columns.Failed plan-safety checks are rewritten or rejected with structured feedback, while SDop targets reasoning-path fragility and unsafe plans.

4 Formal Algorithms and Verification Harness

The verification harness instruments graph pruning and database plan checks while constructing adversarial trap cases for silent-divergence testing.

  • Harness Implementation: The telemetry harness loads enterprise schemas into a NetworkX graph, instruments the pruning router, optionally connects to PostgreSQL, MySQL, or SQL Server, and computes PSC and SDop rates.The artifact includes separate telemetry and baseline runners.
  • Harness Implementation: The harness can run from the repository root with schema, question, hint, engine, and database-connection arguments.The listed command targets PostgreSQL and Tier 1 questions with schema hints.
  • Plan Verification: EXPLAIN, rather than EXPLAIN ANALYZE, is parsed for unsafe sequential scans on configured hot tables anywhere in the plan tree.PostgreSQL and MySQL use different scan-node patterns, while avoiding execution cost and buffer-cache mutation.
  • Silent-Divergence Verification: The trap manifest targets three EX-equivalent reasoning paths that diverge after data shifts: ordering defaults, join type, and NULL-sensitive aggregation.Supplementary instances inject NULLs, orphan rows, or ties to activate each trap.
  • Silent-Divergence Verification: Confirmed silent divergence requires execution match plus either an authored distractor equivalence or an intent violation on a supplementary instance.This operationalizes the distinction between execution success and robust semantic behavior.
  • Silent-Divergence Verification: The native PostgreSQL re-verification covered 132 of 161 manifest entries; 29 Oracle CONNECT BY traps lacked PostgreSQL counterparts, and 36 NOT EXISTS distractors required runtime repair.The repaired distractors were malformed because a templating bug duplicated a SELECT fragment.

5 System Performance Evaluation

DRL’s corrected evaluation shows large context savings and low middleware overhead, while exposing safeguard, loader, and evaluation-harness caveats that constrain interpretation of some metrics.

  • Plan safety: 70.8% gold-SQL PSC for B3 versus 68.6% for B0–B2 indicates a directional safeguard improvement after COALESCE insertion.The B2→B3 comparison is not strictly matched because the COALESCE rewrite excludes some already-safe or non-rewritable statements.
  • Context bounding: −92% context versus naive B0 follows from schema-linking’s −76% reduction plus DRL’s additional −67% B1→B2 reduction.The B1→B2 step reduces mean context from 1,720 to 574 bytes.
  • Cross-vendor verification: 76.3%/77.3% MySQL gold-SQL PSC for B0–B2/B3 remains unchanged after loader correction, while MySQL model-generated PSC remains open.MySQL pruning p95 is 1.67 ms, consistently slower than PostgreSQL’s 0.58 ms, but still sub-2 ms.
  • Failure modes: 254/471 GPT-4o EX failures are semantic/filter errors versus 47/471 invalid-column failures, so pruning cannot resolve wrong join predicates.DRL targets deterministic middleware failures while semantic join errors remain dependent on model behavior.
  • Evaluation validity: 56–60% original executed rates rose to 98.2% for GPT-4o and Gemini and 93.1% for Claude after correcting a shared regex post-processor defect.The defect stripped legitimate table-alias qualifiers, converting correct joins into ambiguous column references and manufacturing a cross-vendor gap.

6 Discussion

DRL frames enterprise NL2SQL as a middleware problem involving bounded context, plan-aware admission, and silent-divergence detection rather than model capability alone. Its router reduces context with low latency, while corrected evaluation and plan inspection expose operational risks that execution match alone misses.

  • Context reduction: 92% context reduction versus B0 is achieved at pruning p95 = 0.58 ms, while DRL’s marginal B1→B2 reduction is 67%.The full B0→B2 reduction includes conventional schema-linking; the B1→B2 figure isolates the router’s contribution.
  • Operational safeguards: 60.8% of flagged EX-passers confirm as true silent divergence, allowing operators to tune admission between rejection and repair.DRL exposes SDop rather than silently returning execution-matching but intent-divergent rows.
  • Dialect portability: PostgreSQL and MySQL agree on pruning and context reduction, while MySQL gold executability is 875/946 because of dialect-translation gaps in the bank.The reported MySQL gap is attributed to the evaluation bank rather than middleware instability.
  • Execution cost: Claude’s maximum observed execution latency reaches 5,048 ms despite comparable mean, p50, and p95 latency across models.The outlier is associated with the correlated-subquery query pattern described elsewhere in the discussion.
  • Plan-aware admission: A correlated subquery can return correct rows yet repeatedly recompute work, making EX-only evaluation insufficient for OLTP deployment decisions.Plan inspection distinguishes a correct-but-quadratic query from a correct-and-index-friendly rewrite; production scale can turn the former into an outage.

7 Related Work

DRL is positioned as an external middleware layer for enterprise NL2SQL, complementing schema linking, prompting, and decoding methods with deterministic context bounds, plan-safety admission, and silent-divergence checks. Its comparison emphasizes a capability conjunction that prior system families were not designed to address together.

  • Benchmarks and evaluation: Prior benchmarks evaluate schemas far smaller than the 465-table catalogs targeted by DRL, while related semantic-evaluation work highlights risks from string matching and test-instance equivalence.Spider, BIRD, and Spider 2.0 remain an order of magnitude smaller in schema scale.
  • Industry NL2SQL middleware: Commercial middleware typically retrieves metadata by vector similarity, while DRL uses deterministic graph traversal with a hard cap and reports safety metrics as first-class evaluation outcomes.The contrasted metrics include RAST compilation, plan safety compliance, and silent-divergence guards alongside execution match.
  • Query plans and database systems: DRL treats plan-safety admission as a middleware release criterion, including sequential scans on hot tables, rather than merely optimizing already-correct fixed queries.This extends classical query-plan analysis into admission control for LLM-generated SQL before shared OLTP connections are reached.
  • Schema linking and structured decoding: Schema-linking encoders provide a soft in-model analogue of context bounding, whereas DRL enforces an external cardinality cap.The comparison marks partial credit where prior systems offer softer or incomplete versions of DRL’s capabilities.
  • Comparative scope: No compared prior system family was designed for the conjunction of OLTP-scale catalogs, plan-safety admission, and silent-divergence detection.The comparison is scoped to the listed system families and does not disparage systems evaluated on smaller schemas or different targets.
  • Positioning: DRL supplies hard context caps and post-generation verification outside generation, so any model or prompting strategy can consume its bounded schema envelope.This positions DRL as complementary to schema-linking encoders and decomposition methods rather than as a competing prompting recipe.

8 Threats to Validity

The evaluation’s validity is constrained by a corrected harness defect, single-run measurements, synthetic-but-realistic schemas, changing model versions, and limitations of execution match and taxonomy counts. These boundaries affect reproducibility, generalization, and interpretation of exact percentages.

  • Internal validity: The harness defect is the most direct internal-validity threat, and middleware latency figures are single-run wall-clock measurements.The artifact retains pre- and post-correction result sets, while the paper avoids treating single-run latency values as more precise than warranted.
  • External validity: The suite uses six synthetic-but-realistic OLTP schemas rather than live production catalogs, so collision and partial-FK rates are design parameters, not population measurements.Specific percentages should not be generalized beyond catalogs sharing the suite’s construction assumptions.
  • External validity: The evaluated models are a snapshot because providers may update weights without changing public identifiers, weakening exact later replication.The paper reports timestamps with released results to document this versioning boundary.
  • Construct validity: EX can both over- and under-estimate correctness, and roughly one-fifth of a taxonomy bucket’s raw count may reflect comparator strictness, gold authoring, or ambiguous phrasing.Exact percentages in Tables 8 and 9 should therefore be treated as directional rather than perfectly accurate.

9 Limitations and Future Work

DRL’s implemented safeguards and pruning router are evaluated, while full serving-path parsing, emission, MySQL model-generated plan safety, and write-path NL2SQL remain outside scope.

  • Implementation scope: The pruning router, EXPLAIN/NULL safeguards, and offline RAST T1/T2 validator are implemented, but full parser and dialect emission remain unintegrated in serving.The offline validator uses sqlglot and has been run on all 1,000 questions’ model-generated SQL.
  • Measurement boundary: B3 measures pruning plus COALESCE/EXPLAIN safeguards, not RAST typing or emission, so its B2→B3 delta cannot isolate RAST’s contribution.T1/T2 violation rates are reported independently.
  • Router limitation: The router’s FK-degree tie-break removes catalog-order fragility but remains heuristic and can rank an irrelevant high-degree reference table first.Embedding similarity is identified as a possible refinement.
  • Plan safety: PSC deltas are directional because B3 uses smaller denominators than B0–B2, preventing a strictly matched-sample comparison.The denominators are n=946 for PostgreSQL and n=832 for MySQL, versus n=1,000/875 for B0–B2.
  • Cross-vendor scope: MySQL model-generated PSC remains open because available SQL is a stale GPT-4o-only subset predating harness correction.The paper declines to compute a number from pre-correction predictions.
  • Deployment scope: DRL enforces read-only admission, leaving write-path NL2SQL unaddressed.This is an explicit scope boundary rather than a claim about write-path performance.

10 Artifact Availability

The paper releases the artifacts needed to reproduce middleware metrics and NL2SQL evaluation, including the verification suite, harnesses, baseline scripts, figures, and RAST validator.

  • Artifact availability: All artifacts needed to reproduce middleware metrics and NL2SQL evaluation are open in the repository.The release includes data, scripts, harnesses, and figure-generation code.
  • Evaluation data and scripts: The repository provides a 1,000-pair verification suite and scripts for PostgreSQL NL2SQL evaluation and confirmed-SD re-verification.The suite is stored in verification_suite_1000.json.
  • Middleware evaluation: Baseline and telemetry scripts expose B0–B3 middleware metrics plus pruning and EXPLAIN measurements.These support reproduction of the reported middleware experiments.
  • Safety and typing: The repository includes model-generated PSC measurement code and RAST T1/T2 typing-validator artifacts.The RAST materials include measure_rast_typing.py and rast_compiler.py.
  • Reproduction conditions: Middleware timings exclude LLM latency, and all three models are evaluated live under the corrected PostgreSQL harness.Claude receives deterministic schema-prefix stripping applied uniformly alongside the join-alias-preserving fix.

11 Threat Model and Operational Deployment

DRL is deployed as a read-only gateway that bounds context, rejects unsafe plans, and repairs nullable aggregates before execution, with explicit testing and regression gates.

  • Threat model: DRL assumes an untrusted LLM front-end and trusted catalog/plan verifier, enforcing read-only admission, context caps, EXPLAIN rejection, and NULL rewriting.The safeguards target DDL/DML, cross-schema access, hot-table scans, and nullable aggregates.
  • Operational protections: DRL rejects or rewrites accidental over-broad queries before they reach the shared connection pool, regardless of semantic correctness.The example is a sequential scan over a hot customers table.
  • Deployment: The per-request path is prune, prompt, generate, sanitize, check SDop/PSC, then execute or repair with at most two attempts.Measured prune-plus-middleware overhead remains below 5 ms p95 excluding the LLM.
  • Operational discipline: The sanitize step belongs on the correctness-critical path and should be versioned and tested like the router and safeguards.A harness defect demonstrates that post-processing can silently corrupt otherwise-correct SQL.
  • Regression testing: CI can gate releases on context reduction, pruning latency, PSC, EX/SDop bands, and direct sanitize-step tests.The proposed thresholds include ≥65% B1→B2 reduction and ≤2 ms prune p95.

12 Conclusion

The paper concludes that enterprise NL2SQL at OLTP scale is an architecture problem requiring context bounding, verification, and plan-aware admission. DRL and its open suite target semantic, structural, and plan-safety failures rather than leaderboard performance alone.

  • Conclusion: 92% context reduction over naive full-catalog prompting, with sub-millisecond pruning, accompanies DRL’s architecture for enterprise NL2SQL.The conclusion also reports 67% marginal reduction over conventional schema-linking.
  • Failure modes: Semantic and structural errors, rather than column hallucination, are identified as dominant failure modes across the 1,000-pair suite.Invalid-column failures remain a minority at 10–12%.
  • Evaluation outcome: GPT-4o, Claude Sonnet 4.5, and Gemini 2.5 Flash cluster around 52–53% execution match with statistically indistinguishable Wilson intervals.The paper frames this pattern as evidence for systems-focused evaluation rather than leaderboard chasing.
  • Evaluation integrity: The corrected harness removed an apparent 4–10 percentage-point cross-vendor gap caused by post-processing, extending scrutiny to benchmark code.The defect silently suppressed measured accuracy for all three models.
  • Operational policy: The production prompt enforces a strict pruned-schema boundary, read-only single-statement SQL, requested-column projection, NULL safety, and selective predicates.It also constrains joins to FK edges in the pruned schema and requires plan-safety checks before emission.
  • SQL safeguards: The prompt requires COALESCE handling for nullable aggregates and ordering, join, filter, flag, and pagination rules tailored to the target dialect.These rules include NULLS LAST for DESC ordering unless NULLS FIRST is requested and ANSI FETCH FIRST pagination.
  • Pre-emission verification: Before emission, DRL checks column existence, grouping completeness, and sequential-scan risk.These checks operationalize schema validity, aggregate correctness, and selective predicate placement.

B Algorithm 1 Worked Trace

The worked trace shows DRL selecting a conservatively bounded table subgraph through lexical scoring, degree-based tie-breaking, and capped foreign-key expansion. The corrected router selects five tables in 0.75 ms, including the tables needed for the gold join while retaining plausible but unused candidates.

  • Tokenization: The tokenizer preserves function words and treats policy_level as one token, so it matches only tables whose column lists contain that literal token.Table names are additionally normalized by replacing underscores with spaces before tokenization.
  • Phase 1 scoring: Raw lexical scoring gives policies a score of 2, while nine other tables tie at score 1.The trace applies no domain-hint boost.
  • Phase 1 seed selection: The corrected tie-break ranks equal-scoring candidates by foreign-key degree rather than catalog iteration order.This favors structurally connected tables over alphabetically or file-order-selected alternatives.
  • Phase 2 join closure and truncation: One-hop foreign-key expansion ranks neighbors by degree before applying the k=5 cap, adding customers and encounters and reaching the cap exactly.The resulting trace lists group_policies and policies among the selected closure candidates.
  • Runtime: The router completes this trace in 0.75 ms, consistent with the reported sub-millisecond overhead.
  • Selected context: Vq contains 5 of 412 pooled catalog tables, including policies and claims required by the correct join.Customers, encounters, and group_policies are also selected but unused by the gold query.

C RAST Typing Rules

The appendix makes RAST validation operational by assigning structural output schemas and checking column-resolution and grouping invariants before SQL emission. An offline sqlglot-based validator implements these checks, but broad gating remains limited by substantial false-positive rates.

  • Schema typing: Each RAST node τ receives a structural output schema sch(τ), represented as column-name and nullability pairs.
  • T1 column resolution: T1 requires every referenced column to appear in its immediate RAST child schema, catching invalid-column-reference errors before emission.
  • T2 grouping validity: T2 requires every non-aggregated projection column in Project(Agg(...)) to appear in the grouping key, preventing malformed aggregation queries from reaching the database.
  • NULL handling: COALESCE insertion is restricted to columns capable of containing NULL rather than applied defensively to every aggregate.
  • Implementation: The offline validator uses sqlglot to resolve T1 against the pruned schema and check T2 by walking the concrete expression tree.This closes part of the previously specified-but-not-integrated validation gap.
  • Empirical validation: Across all 1,000 predictions, T1 flags 134/1,000 GPT-4o, 237/1,000 Claude, and 126/997 Gemini predictions, mostly as false positives.The broad validator is therefore less precise as a general-purpose gate than its failure-count agreement suggests.
Loading 2608.26172v1…