PaperlayerView exact revision on arXiv

Source-linked AI summary

Bounded Semantic Planning and Deterministic Compilation for Reliable Enterprise Text-to-SQL

Yi Ai

arXiv:2608.16663v1cs.DBcs.AI

TL;DR

Enterprise text-to-SQL can produce executable queries with incorrect business interpretations. This paper evaluates semantic path compilation, which constrains semantic choices and delegates relational realization to code, finding higher correctness than direct DDL-to-SQL generation on the paired benchmark.

  • Problem

    Enterprise text-to-SQL must resolve relationship paths, business roles, metrics, and aggregation grain, because executable SQL can still yield plausible but incorrect results.

  • Method

    Semantic path compilation combines a finite question-specific semantic choice set, engine-owned relational realization, explicit refusals, and deterministic checking.

  • Results

    37/38 questions achieved solid success with SPC versus 21/38 for direct DDL-to-SQL generation, with no wrong-but-executed SPC run across 114 outcomes.

  • Takeaways & Limitations

    Within this benchmark, the combined SPC design operationally distinguishes executable wrong results from explicit refusals, although component-level contributions remain unestimated.

  • Takeaways & Limitations

    The headline success measure depends on an authored, model-dependent judge whose nontrivial verdicts lacked independent second-human adjudication.

Abstract

from arXiv · show

Direct text-to-SQL asks a language model to do two jobs: interpret the business question and construct the complete relational query. In enterprise schemas, SQL can execute successfully while using the wrong relationship role or aggregation grain. We study an alternative placement of the stochastic boundary. A multi-turn planner grounds phrases and selects from question-specific governed options; graph traversal, role predicates, grain lowering, SQL construction, and deterministic checks are implemented in code. We evaluate this semantic path compilation (SPC) system against direct DDL-to-SQL generation on the ACME insurance benchmark. On a 38-question adjudicated comparison set with three runs per question, SPC was adjudicated correct on every run for 37 questions (97.4%), compared with 21 (55.3%) for the baseline. The paired discordance was 16 questions in favor of SPC and none in favor of the baseline (two-sided exact McNemar p=3.05x10^-5). SPC answered all 38 questions correctly at least once and produced one refusal and no adjudicated wrong-but-executed run across 114 run outcomes; the baseline produced 29 adjudicated wrong runs and seven additional judge-flagged data-only coincidences on the same set. A strict-equivalence sensitivity analysis increased the paired difference. Additional SPC runs with GPT-5.4 and Gemini-3.6-Flash showed similar question-level robustness, although their per-run verdict artifacts were not preserved. Six additional benchmark items are retained in an all-item analysis and documented separately by failure class. The study supports an end-to-end systems result, not a causal claim that compilation alone produced the gain, because SPC receives governed semantic artifacts that the DDL baseline does not.

1. Introduction

Enterprise text-to-SQL must resolve business meaning, relationship roles, and aggregation grain—not merely generate executable SQL. This paper evaluates semantic path compilation, which bounds LLM behavior to grounding and governed selection while deterministic code realizes and checks the relational query.

  • Motivation: Executable SQL and plausible numbers can still reflect the wrong business interpretation, relationship role, relational path, or aggregation grain.Parser validity and agreement on one database instance are insufficient safeguards.
  • Approach: SPC places the stochastic boundary after language grounding and governed selection, then uses deterministic compilation for paths, roles, grain lowering, SQL construction, and governance checks.Typed retrieval exposes a finite, question-specific choice set rather than allowing the model to author relational structure.
  • Approach: The model selects grounded semantic alternatives and cannot name tables, columns, join predicates, role predicates, or SQL fragments.Rejected attempts may be retried once with only the refusal reason carried into a fresh conversation.
  • Contributions: The paper contributes bounded semantic planning, deterministic relational realization, and multiplicity-focused evaluation using counterfactual database instances.The compiler produces complete SQL with governed paths, role predicates, grouping, and supported grain transformations.
  • Scope and limitations: The study compares complete systems in one domain, with a compiler developed while inspecting the benchmark and semantic knowledge not held constant between arms.Accordingly, the evaluation concerns the end-to-end systems rather than an isolated causal effect of compilation.

2. Background and Related Work

Prior work combines ontology-level interpretation, semantic-layer modeling, intermediate representations, constrained decoding, and deterministic SQL translation. Related database and reliability research motivates SPC’s attention to aggregation grain, semantic equivalence, abstention, and question-level stability.

  • Ontology and semantic translation: Ontology-based systems construct ontology-level paths or queries before deterministic translation to SQL, including ATHENA, ATHENA++, and Ontop.ATHENA uses ranked interpretation trees; ATHENA++ extends the approach to nested and aggregation queries; Ontop rewrites SPARQL over OWL 2 QL mappings.
  • Semantic layers: Commercial semantic layers model entities and compose joins within a governed semantic surface, while related studies examine semantic context and knowledge-graph querying.MetricFlow generates SQL within its modeled semantic surface; the cited studies provide external context on the ACME benchmark, semantic context, and ontology-based querying.
  • Constrained generation and compilation: IRNet, PICARD, and Semantic Model Query compilation reduce SQL search or move construction behind an intermediate representation; SPC instead limits planner freedom to governed vocabulary and routing.IRNet lowers SemQL to SQL, PICARD rejects inadmissible continuations through incremental parsing, and SPC differs in where remaining model freedom is placed.
  • Database semantics and reliability: Prior work frames aggregation placement, summarizability, and grain propagation as database problems, while reliability studies motivate reporting refusals and wrong-but-executed answers separately.The cited database literature addresses eager aggregation, summarizability, heterogeneous multidimensional schemas, and grain propagation; reliability work supports explicit abstention and question-level stability analysis.
  • Semantic evaluation: Execution-correct SQL can conceal semantic differences on one database instance, motivating counterfactual testing focused on role- and multiplicity-sensitive enterprise errors.The cited approach follows distilled multi-instance evaluation in principle but targets enterprise relationship roles and multiplicity rather than introducing a new semantic-equivalence criterion.

3. Problem Definition

The task is to produce SQL that answers a natural-language analytical question over an enterprise database according to governed business semantics. The problem is that direct generation leaves relational decisions implicit, whereas SPC exposes bounded semantic choices and deterministic transformations without proving business correctness.

  • Task definition: The task is to produce SQL whose returned rows answer a natural-language question over an enterprise database according to its semantic metadata.Inputs include the question, database, and semantic metadata describing business concepts, metrics, attributes, and relationships.
  • Problem: Direct generation must jointly choose relationship paths, business roles, metric interpretations, aggregation grain, filters, attachment points, and SQL realization.These choices may be relationally valid yet semantically inappropriate for the question.
  • SPC boundary: SPC makes a subset of relational decisions explicit as typed choices and applies deterministic transformations to the resulting plan.Its purpose is to govern decisions that are implicit in sampled direct-generation output.
  • Outcome definitions: The evaluation defines task success by whether returned rows satisfy the question under the adjudication rule, distinguishing solid success, at-least-once success, refusals, and wrong-but-executed outcomes.Solid success requires all three runs for a question to succeed; at-least-once success requires one or more.
  • Execution contract: The SPC contract bounds model naming and makes relational realization reproducible through deterministic retrieval, compilation, and checking after stochastic planning.This contract is not a proof of business correctness because undecidable findings are non-blocking and the planner may choose the wrong admissible interpretation.

4. Proposed Method

SPC places the stochastic boundary in bounded semantic planning: the planner resolves question-specific governed options, while deterministic code enumerates routes, compiles plans, and checks emitted SQL. For fixed inputs, compilation is repeatable, but correctness still depends on planner choices and incomplete checker coverage.

  • Planner–engine boundary: The planner uses a fixed typed workflow without shell or SQL access, calling retrieval or review tools before submitting a structured semantic plan.The planner’s ownership boundary is separated from the engine that constructs and validates the submission schema.
  • Planner–engine boundary: Question-specific schemas enumerate only retrieved subjects, metrics, attributes, operators, and route handles, preventing ordinary submissions from naming unretrieved objects.The engine independently validates semantic completeness because schema conformance alone is insufficient.
  • Governed semantic model: Roles are part of edge identity rather than generic traversal filters, so Party-to-Policy routes commit to business roles such as PH, AG, or UW.Longer routes through role objects preserve the same bound party-role predicate, preventing role-free arrival at Policy.
  • Deterministic route resolution: SPC deterministically enumerates governed, role-valid, supported routes under a hop bound and exposes path-opaque handles, leaving the planner only to select among returned alternatives.This replaces open-ended edge-sequence synthesis with bounded semantic choice; handles identify routes without giving the model an editable edge chain.
  • Compilation and checking: The checker deterministically evaluates join coverage, role predicates, metric definitions, literal grounding, and supported grain patterns, blocking violation findings but allowing undecidable findings.The guarantee is conditional: fixed plan, ontology, mapping, and compiler version yield the same SQL, without guaranteeing planner correctness or checker completeness.

5. Experimental Design

The study compares direct DDL-to-SQL generation with SPC on a fixed ACME insurance benchmark, using three runs per question and adjudicated question-level solid success as the primary estimand. The design includes targeted counterfactual tests and documents limitations in sampling parity, exclusions, generalization, and end-to-end reproducibility.

  • Counterfactual evaluation: Counterfactual databases were designed to preserve query-relevant business facts while changing unconstrained relational multiplicities, separating grain-unsafe implementations from grain-correct references.These were targeted tests of enumerated failure modes, not independent samples from a population of enterprise databases.
  • Experimental arms: The paired experiment compares B0, which generates SQL directly from the question and DDL, with SPC, which selects governed semantic plans and uses deterministic relational realization.Because the arms differ in both information and mechanism, the comparison estimates an end-to-end system effect rather than compilation in isolation.
  • Protocol and estimand: Three runs per question were conducted with gpt-5.6-luna, and primary correctness used question-level solid success, requiring all three runs to be accepted.The paired hypothesis test was a two-sided exact McNemar test on question-level solid success.
  • Benchmark and exclusions: 38 questions formed the primary paired set after six items were uniformly removed under a fixed registry.The registry was constructed post hoc during reference auditing, then frozen before rescoring and table regeneration; it was not preregistered.
  • Limitations: The primary comparison does not establish cross-domain generalization because the questions are not a random sample of enterprise workloads.Primary aggregate statistics are auditable from frozen verdict artifacts, but the study is not currently reproducible end to end because execution and campaign-state artifacts are unavailable.

6. Results

On the 38-question paired set, SPC substantially outperformed B0 under inclusive and strict scoring, while additional planner models showed similar question-level results. The findings concern the complete SPC system and relocate, rather than eliminate, stochasticity.

  • Primary results: 97.4% (37/38) of questions achieved solid success with SPC, versus 55.3% (21/38) for B0 under the frozen inclusive rule.At least-once success was 100% (38/38) for SPC and 81.6% (31/38) for B0.
  • Primary results: 113/114 SPC runs succeeded and none were adjudicated wrong, compared with 78/114 successful and 29/114 adjudicated wrong B0 runs.B0 also produced seven data-only divergence flags, whereas SPC produced one no_sql refusal.
  • Primary results: 42.1 percentage points separated solid success, with 16 paired questions favoring SPC and none favoring B0.The paired McNemar comparison applies specifically to these 38 benchmark questions, not a random population of enterprise queries.
  • Sensitivity analyses: 97.4% (37/38) SPC solid success persisted under strict scoring, while B0 fell to 47.4% (18/38).The strict paired table contained 19 SPC-only solid questions and no B0-only solid questions.
  • Planner robustness: 97.4% (37/38), 94.7% (36/38), and 92.1% (35/38) were the solid-success rates for gpt-5.6-luna, gpt-5.4, and gemini-3.6-flash, respectively.These results support robustness to tested planner choice, but model effects are not isolated because only Luna has a paired B0 comparison.

7. Discussion

SPC’s operational value comes from bounded semantic choices, engine-owned relational realization, and explicit refusals rather than deterministic SQL rendering alone. Its evidence remains limited to the ACME scope, represented business compositions, and benchmark data capable of exposing known hazards.

  • Evaluation scope: The implementation has no frozen held-out evaluation of unseen construct combinations, so the paper makes no generalization claim beyond ACME or workload-independent scaling claim.The components were developed while inspecting this benchmark, despite being organized around reusable relational constructs.
  • Design and operational role: SPC combines finite question-specific semantic choices, engine-owned relational realization, and explicit refusal paths; the experiment does not isolate each component’s contribution.An executable wrong result and a refusal are operationally different outcomes.
  • Coverage boundary: If an intended business composition lacks an admissible route, SPC refuses rather than inferring or approving it from physical table connectivity.The semantic model or physical mapping must be extended before the query can be accepted.
  • Counterfactual analysis: Static rules identify known hazards, but benchmark multiplicities must make those hazards observable; discriminating data do not prove that the reference reflects intended business meaning.Such data establish separability from enumerated alternatives only.

8. Practicality

SPC shifts substantial effort to semantic modeling, compiler implementation, testing, and governance, making deterministic compilation most attractive for stable, complex, and high-risk enterprise query domains. It is therefore a hybrid trade-off rather than a universal replacement for direct LLM SQL generation.

  • When direct generation fits: Direct LLM SQL generation may offer a better trade-off for ad hoc, lightly modeled domains with predominantly simple analytical questions.It requires less up-front modeling and can accommodate previously unseen query structures without extending a compiler.
  • When SPC fits: Deterministic compilation is more attractive when business semantics are stable, queries reuse deep or role-sensitive relational constructs, and plausible errors carry operational cost.These conditions favor repeatability and explicit refusal over unrestricted query flexibility.
  • Cost trade-off: SPC trades development-time modeling, compiler, testing, and governance costs for reduced stochastic relational reasoning at query time.The architecture is explicitly not presented as a universal replacement for conventional semantic-layer or text-to-SQL systems.
  • Scalability assumption: Generic lowering rules are intended to make compiler complexity grow by reusable relational construct rather than by individual benchmark question.The held-out evaluation needed to test this distinction directly is not reported.
  • Deployment model: The expected deployment model is hybrid: governed queries use deterministic compilation, while unsupported semantics may be clarified, refused, or sent to a less constrained fallback.Fallback behavior depends on the application’s risk requirements.

9. Threats to Validity

The evaluation’s validity is limited by its single-domain scope, limited model diversity, benchmark exposure and development, researcher involvement, and authored adjudication. The comparison also cannot isolate compilation effects from governed semantic knowledge or protocol differences, while checker and artifact limitations constrain broader guarantees and reproduction.

  • Scope and model diversity: Results are limited to ACME Insurance, and three planner configurations across two model families provide only limited evidence of invariance across providers, families, or future versions.Only Luna has a paired baseline, and the study does not claim cross-domain generality.
  • Benchmark-related threats: Benchmark exposure and development may inflate absolute success rates or encode benchmark-specific behavior, and no held-out construct-combination result is reported.The benchmark may have appeared in training data, while compiler, resolver, and guardrail improvements were made during benchmark examination.
  • Researcher and adjudication effects: Author involvement in system, ontology, compiler, counterfactual, infrastructure, and adjudication design creates implementation and adjudication bias despite stated mitigations.Mitigations include frozen verdict artifacts, uniform registry application, independent reference computations, strict-scoring sensitivity, and original-denominator reporting.
  • Evaluation validity: The authored, model-dependent judge and selected reference-answer corrections limit interpretation of adjudicated success, including structural data-only judgments and execution-port incompatibilities.The judge uses SQL, rows, deterministic short-circuits, up to three votes, and a failure default, but agrees_on_this_data_only is not a mechanically demonstrated counterfactual failure.
  • Attribution, guarantees, and reproducibility: The B0–SPC comparison estimates an end-to-end system effect rather than compiler causality because SPC receives governed semantic knowledge and has a different refusal-retry protocol.Ten of 114 primary SPC outcomes invoked the retry, nine producing SQL; the checker also leaves undecidable findings unblocked, and missing databases and environments prevent end-to-end reproduction.

10. Conclusion

On the 38-question ACME set, SPC achieved solid success on 37 questions versus 21 for direct DDL-to-SQL generation, with no wrong-but-executed run across 114 outcomes. The result supports higher repeated-run reliability for this governed workload, but not a causal claim about deterministic compilation alone or transferability beyond the benchmark.

  • Results: 37 of 38 questions were solidly answered by SPC, compared with 21 of 38 for direct DDL-to-SQL generation.The comparison used the paired ACME benchmark set.
  • Results: 0 wrong-but-executed runs occurred in 114 SPC run outcomes.Strict-scoring sensitivity strengthened rather than weakened the difference.
  • System interpretation: SPC moved path, role, grain, and SQL realization into explicit deterministic machinery while preserving answer coverage for the governed workload.The language model selected governed semantic objects but did not author relational structure or SQL.
  • Limitations: The findings are an end-to-end systems result on one benchmark, not proof that deterministic compilation alone caused the gain or that results transfer across domains and models.The study also does not establish that the checker is sound for arbitrary SQL.

Code and Data Availability

The authors release a partial MIT-licensed implementation and semantic model, but omit core evaluation materials, so the reported measurements cannot presently be reproduced from the public snapshot.

  • Released artifacts: The MIT-licensed snapshot includes the semantic planner, path resolver, grain-aware compiler, static checker, ACME semantic model, and physical mapping.It is available at github.com/yai333/Semantic-Planning-and-Deterministic-for-Text-to-SQL.
  • Released artifacts: The snapshot excludes benchmark questions, gold labels, database fixtures, evaluation harnesses, frozen verdict artifacts, and execution traces.
  • Reproducibility limits: The reported aggregate statistics were regenerated from frozen verdict files that are not included in the public snapshot.
  • Reproducibility limits: The execution databases and exact working-tree state of the primary campaign are unpublished, so the reported measurements cannot presently be reproduced.The published compiler and checker implementations can still be inspected and exercised from the public code snapshot.

Authoring and Tool Disclosure

Generative AI tools supported software development and manuscript editing, while the authors retained responsibility for the study’s design, provenance, verification, analysis, and claims.

  • Generative AI tools were used during software development and manuscript editing.
  • The authors remained responsible for the experimental design and code artifact provenance.
  • The authors also retained responsibility for literature verification, statistical analysis, and all paper claims.
Loading 2608.16663v1…