PaperlayerView exact revision on arXiv

Source-linked AI summary

SemPlan: Benchmarking Structured Semantic Planning for LLM-Based Queries over Enterprise Data

Bruno Santos Teixeira

arXiv:2608.13612v1cs.AIcs.SE

TL;DR

Enterprise-data interfaces must resolve semantic, policy, and operational constraints beyond producing valid SQL. SemPlan compares four LLM–software responsibility splits under a frozen bilingual synthetic benchmark, finding no universal winner across correctness, policy behavior, cost, and failure modes.

  • Problem

    Enterprise-data queries require resolving semantics, authorization, temporal scope, and conversational context, so syntactic SQL validity alone is insufficient.

  • Method

    SemPlan evaluates four LLM–software responsibility splits using a controlled bilingual benchmark and fixed model, cases, database snapshot, and evaluation rules.

  • Results

    A3 had the highest observed answer correctness and repeatability, A4 the lowest mean API cost and false-refusal rate, and A1 the strongest policy-correct behavior.

  • Takeaways & Limitations

    Architectural evaluation should consider trade-offs across multiple dimensions rather than rank systems by a single accuracy metric.

  • Takeaways & Limitations

    Generalization beyond one synthetic enterprise domain, model/provider configuration, database system, and two languages remains unknown.

Abstract

from arXiv · show

Natural-language interfaces to enterprise data must translate underspecified requests into governed, executable behavior while controlling invalid queries, policy failures, cost, and nondeterminism. SemPlan Benchmark evaluates this architectural design space with a deterministic synthetic bilingual benchmark containing 1,800 cases in English and Brazilian Portuguese; 1,200 cases form the frozen scientific evaluation subset. Four architectures are compared under the same model configuration: direct SQL generation (A1), a bounded tool-agent baseline (A2), structured semantic-request generation followed by deterministic planning and execution (A3), and a clarification/stateful semantic-plan variant (A4). Across 4,800 primary records, answer correctness was low in absolute terms: 22.25% for A1, 22.58% for A2, 25.67% for A3, and 24.25% for A4. A3 had the highest observed correctness and significantly exceeded A1, A2, and A4 in the pre-specified paired correctness analysis, while A1 retained the highest policy-correct rate and the lowest unsafe-or-invalid rate. A4 had the lowest mean API cost and lowest false-refusal rate. On a preselected 150-case stability subset, answer-correct repeatability ranged from 92.00% to 98.67%. The results support a trade-off interpretation rather than a universal ranking: additional structural constraints changed failure modes and efficiency, but did not monotonically improve correctness or solve ambiguity and multi-turn state consistency.

1 Introduction

The introduction frames enterprise natural-language data access as a semantic, policy, operational, and conversational reliability problem rather than merely an SQL-generation task. SemPlan evaluates how structural constraints across four architectures trade off correctness, policy behavior, failure modes, cost, ambiguity, state consistency, and repeatability under controlled conditions.

  • Motivation: Enterprise questions can involve business metrics, implicit dimensions, temporal scopes, authorization boundaries, unsupported actions, and conversational context, making syntactically valid SQL potentially semantically wrong or policy-inconsistent.The introduction emphasizes that operational validity extends beyond SQL syntax.
  • Architectural design space: LLM-based systems distribute responsibility among direct SQL generation, tool selection, typed semantic representations, and deterministic normalization or execution layers.These architecture choices define hypotheses about where errors arise and which constraints improve reliability.
  • Study scope: SemPlan holds the model, benchmark cases, database snapshot, and evaluation rules fixed to study trade-offs among structural constraint, correctness, policy behavior, failure modes, cost, ambiguity, state consistency, and repeatability.The study is explicitly narrower than a claim of production readiness.
  • Contributions: The paper introduces a bilingual synthetic benchmark, compares four architecture patterns under a common configuration, and reports paired evidence across correctness, policy behavior, typed failures, cost, ambiguity, state consistency, and repeatability.The benchmark uses an independently generated synthetic enterprise domain rather than customer or proprietary data.

2 Related Work · 3 Research Questions and Pre-Specified Hypotheses

SemPlan is situated among grammar-based, intermediate-representation, constrained-decoding, and tool-using approaches to natural-language database interfaces, while separating benchmark-generation and evaluation artifacts for reproducibility. Its frozen study tests five questions comparing A1–A4 on correctness, failure outcomes, efficiency, ambiguity handling, and multi-turn consistency against pre-specified directional hypotheses.

  • 2 Related Work: Natural-language database interfaces historically used grammar, parsing, semantic representations, domain constraints, and schema-specific transformations before modern LLM systems.
  • 2 Related Work: Intermediate structures and constrained decoding narrow the gap between language and executable SQL through schema linking, semantic representations, execution feedback, or token-level admissibility.Examples include IRNet, RAT-SQL, execution-guided decoding, and PICARD.
  • 2 Related Work: Tool-using systems form another architecture family, combining interleaved reasoning and actions, learned API invocation, or schema-constrained structured outputs and function calls.SemPlan is presented as complementary because it evaluates an architectural design space rather than proposing a single decoding technique.
  • 2 Related Work: SemPlan separates synthetic data generation, benchmark generation, human review, gold execution, model execution, and derived statistical artifacts to support reproducibility.This follows the principle that datasets should communicate motivation, composition, generation, intended use, and limitations.
  • 3 Research Questions and Pre-Specified Hypotheses: Five frozen research questions compare A1–A4 on bilingual answer correctness, invalid or failed execution and policy outcomes, cost and latency, ambiguity clarification, and PATCH/REPLACE follow-up consistency.
  • 3 Research Questions and Pre-Specified Hypotheses: The pre-specified hypotheses expected semantic-plan approaches to improve correctness, reduce invalid or policy-violating execution, and improve ambiguity and multi-turn consistency.They also expected A3 cost to remain at or below the tool-agent approach, with A4 clarification and structured state supporting ambiguity and follow-up behavior.
  • 3 Research Questions and Pre-Specified Hypotheses: The analysis preserves mixed and negative findings rather than redefining the hypotheses after observing results.

4 SemPlan Benchmark

SemPlan is a deterministic synthetic bilingual benchmark for governed enterprise-data queries, built from parameterized semantic cases and gold plans executed against a read-only PostgreSQL reference environment. Its 1,800-case release includes a frozen 1,200-case scientific evaluation subset, with development and validation excluded from primary comparison.

  • Reference environment: The synthetic Northstar Commerce domain covers customers, products, orders, payments, expenses, budgets, suppliers, contracts, and calendar data under deterministic generation and governed read-only PostgreSQL views.Monetary values, dates, status fields, and derived business metrics follow documented invariants.
  • Case construction: Parameterized semantic templates cover intents, metrics, dimensions, filters, temporal contexts, grouping, sorting, limits, ambiguity conditions, and policy expectations, with English and Brazilian-Portuguese surface forms.Cases are validated for canonical identifiers and language quality, and gold semantic plans execute deterministically against the reference database to produce normalized gold answers.
  • Benchmark composition: 1,800 cases comprise 900 en-US and 900 pt-BR examples across development, validation, public-test, hidden-test, multi-turn, and adversarial splits.The frozen scientific subset contains 1,200 cases from the public test, hidden test, multi-turn, and adversarial splits.
  • Benchmark composition: 1,200 cases form the frozen scientific evaluation subset, while development and validation cases are excluded from the primary scientific comparison.The release manifest records 3,600 approved case/gold review records from a single human reviewer, treated as a validity limitation.

5 Evaluated Approaches · 6 Experimental Setup

The comparison spans four increasingly structured approaches, from guarded direct SQL and bounded tools to deterministic semantic planning with optional clarification state. The frozen evaluation uses 1,200 scientific cases per approach, repeated stability testing, and paired statistical analyses, while excluding latency from primary conclusions because typed failures recorded zero latency.

  • 5 Evaluated Approaches: A1 permits model-authored SQL, A2 restricts actions to typed tools, and A3–A4 emit typed semantic requests processed by deterministic software.A4 additionally exposes structured state and clarification behavior.
  • 5 Evaluated Approaches: A1-generated SQL passes parsing, AST allowlists, complexity, authorization, timeout, and row-limit checks before execution.A1 receives no gold SQL, plans, or answers.
  • 5 Evaluated Approaches: A2 uses canonical typed-tool schemas, bounded composition, and deterministic implementations without arbitrary SQL or invented tools.Its tools cover operations including aggregation, ranking, period comparison, budget comparison, contract status, and field description.
  • 5 Evaluated Approaches: A3 emits a strict semantic-request envelope that deterministic normalization validates and compiles into an executable semantic plan.Governed software then produces parameterized SQL or an equivalent deterministic operator call.
  • 5 Evaluated Approaches: A4 extends A3 with prior structured state, typed clarification outcomes, and explicit PATCH/REPLACE semantics for multi-turn requests.Only explicitly changed PATCH fields update, while out-of-scope turns preserve state unless reset.
  • 6 Experimental Setup: 4,800 primary records come from 1,200 scientific cases executed once under each approach using the same case set, database snapshot, model configuration, and frozen prompts.A 150-case subset receives two additional executions per approach, producing 1,200 stability records without inflating the primary sample.
  • 6 Experimental Setup: Paired binary comparisons use McNemar tests, paired risk differences with 95% confidence intervals, and Holm-Bonferroni adjustment, while API cost uses paired bootstrap contrasts.Provider, transport, infrastructure, and model-quality failures are distinguished before interpretation.
  • 6 Experimental Setup: Latency is excluded from primary conclusions because typed failures recorded zero latency and were unevenly distributed across approaches.Token usage, provider cost, and failure counts remain reportable from independent ledgers.

7 Results

Absolute answer correctness remained modest across approaches, with A3 highest and significantly above the alternatives in paired analysis. Policy outcomes differed: A1 was safest by policy metrics, while A4 minimized false refusals but had the highest unsafe-or-invalid rate.

  • Primary answer correctness: 25.67% was A3’s highest observed answer-correctness rate, followed by A4 at 24.25%, A2 at 22.58%, and A1 at 22.25%.Absolute answer correctness was low across all approaches.
  • Primary answer correctness: +3.42 percentage points was A3 over A1, and +3.08 points was A3 over A2 in paired answer-correct analysis.Both contrasts had Holm-adjusted p = 0.000110; A4 was 1.42 points below A3.
  • Policy behavior: 43.67% was A1’s highest policy-correct rate, while 31.00% was its lowest unsafe-or-invalid rate.A3 and A4 were both below A1 on policy correctness, so A3’s correctness advantage did not establish a uniformly safer architecture.
  • Policy behavior: 0.17% was A4’s lowest false-refusal rate, but A4 also had the highest unsafe-or-invalid rate at 64.08%.A3 recorded 37.33% policy correctness and 60.58% unsafe-or-invalid.
  • Language-stratified correctness: 31.00% versus 20.33% was A3’s answer correctness in en-US versus pt-BR, compared with A4 at 29.33% versus 19.17%.Correctness was consistently higher in en-US than pt-BR for all four approaches; subgroup results were descriptive/exploratory.

8 Failure Analysis

Failure analysis treats typed model-quality errors as central scientific outcomes rather than missing data. The approaches shifted failure surfaces: A1 was conservative with fewer terminal errors but many more false refusals, while structured interfaces exposed different contract and execution failures.

  • Typed failure categories: Across all 6,000 scientific records, EXECUTION_FAILED (1,762) was the most frequent recorded error, followed by OUTPUT_SCHEMA_INVALID (1,051) and POLICY_VIOLATION (368).The remaining counts were CATALOG_UNKNOWN_ID (78) and CFG_INVALID (24).
  • Typed failure categories: EXECUTION_FAILED denotes governed-execution failure, OUTPUT_SCHEMA_INVALID strict-contract failure, POLICY_VIOLATION guard rejection, CATALOG_UNKNOWN_ID unknown identifiers, and CFG_INVALID configuration or contract invalidity.
  • Approach-level outcomes: A1 produced 372 primary ERROR outcomes versus 747 for A2, 727 for A3, and 769 for A4, but produced 257 false refusals versus 22, 19, and 2.This pattern supports a failure-shift interpretation in which direct SQL with strict guards and typed semantic/tool interfaces expose different failure surfaces.

9 Stability, Ambiguity, and Multi-Turn Behavior · 10 Cost and Operational Results

Across stability, ambiguity, multi-turn, and cost tests, structured approaches showed selective gains rather than universal superiority. A3 was most repeatable and correct on clarification decisions, while A4 minimized API cost, but conversational robustness remained weak overall.

  • 9 Stability, Ambiguity, and Multi-Turn Behavior: 98.67% was A3’s highest observed answer-correct repeatability, compared with 98.00% for A1 and A4 and 92.00% for A2.The stability substudy covered 150 preselected cases with three independent executions per approach and did not claim full determinism.
  • 9 Stability, Ambiguity, and Multi-Turn Behavior: 27.38% was A3’s clarification-decision correctness over 84 ambiguity cases, versus 26.19% for A2, 8.33% for A4, and 0.00% for A1.The clarification-enabled A4 design did not validate the expectation that structured clarification would dominate the alternatives.
  • 9 Stability, Ambiguity, and Multi-Turn Behavior: 20.00% was A1’s multi-turn sequence-state correctness over 200 cases, compared with 17.00% for A2, 14.00% for A3, and 14.50% for A4.Structured state did not improve sequence-state correctness in this evaluation.
  • 10 Cost and Operational Results: USD 0.000469 was A4’s mean API cost per primary record, followed by USD 0.000512 for A3, USD 0.000918 for A1, and 0.000927 for A2.These are provider-API observations under the frozen model and price table, not total cost of ownership.
  • 10 Cost and Operational Results: -USD 0.000459 was A4’s paired mean cost difference versus A2, while A3’s corresponding difference versus A2 was -USD 0.000415.A4 was the least expensive observed approach, and A3 was also substantially cheaper than A1/A2 under the frozen model and price table.
  • 10 Cost and Operational Results: USD 4.028973 was the complete F7 scientific run’s recorded API charge.The reported charges exclude some components of total cost of ownership.
  • 10 Cost and Operational Results: 8,120,408 input tokens, 1,179,317 cached input tokens, 1,540,985 output tokens, and 639,950 reasoning tokens were recorded for provider usage.These counts support reproducibility and budget analysis but should not be generalized to other models or future pricing.

11 Discussion

The discussion finds no monotonic “more structure is better” relationship: A3 leads observed correctness and repeatability, while A1 and A4 retain distinct policy and efficiency advantages. All approaches perform worse on pt-BR than en-US, and even the best architecture is correct on roughly one quarter of primary cases, framing SemPlan as comparative trade-off evidence rather than deployment certification.

  • Architectural trade-offs: A3 occupies the strongest observed correctness/repeatability position, but its policy behavior is weaker and its unsafe-or-invalid rate remains high.A3 also has a mean cost well below A1/A2.
  • Architectural trade-offs: A4 achieves the lowest mean API cost and almost eliminates false refusals, but structured state and clarification introduce additional opportunities for inconsistency.These capabilities may improve governance while also creating more failure opportunities.
  • Architectural trade-offs: A1 has the best policy-correct rate and lowest unsafe-or-invalid rate despite comparatively weak answer correctness and false-refusal performance.A2 provides only a small raw correctness increase over A1, with the weakest repeatability and a large error surface.
  • Multilingual results: Every approach is less accurate on pt-BR than en-US in the frozen primary set despite balanced language coverage and a passed language-quality gate.The discussion attributes the gap to possible differences in surface-form difficulty, lexical mappings, model behavior, or their interactions.
  • Limitations: Roughly one quarter is the best architecture’s correctness level across the benchmark’s difficult combined distribution, so SemPlan evaluates comparative trade-offs rather than deployment readiness.The distribution combines clean, adversarial, multi-turn, and policy-sensitive cases.

12 Threats to Validity

The study’s conclusions are constrained by implementation and measurement choices, limited generalizability, statistical caveats, and author involvement. These limitations qualify comparisons without negating the benchmark’s controlled design and reproducibility measures.

  • Internal validity: Prompt design, implementation details, provider nondeterminism, retry behavior, and the F7 execution hotfix lineage can affect architecture comparisons.The hotfix converted database crashes into typed scientific outcomes without changing benchmark semantics or the underlying model/approach failure.
  • Construct validity: Answer correctness measures equivalence to a normalized canonical result, not complete user usefulness, explanation quality, business trust, or all governance concerns.Repeatability measures binary correctness stability across three executions on a subset and is not equivalent to deterministic generation; latency is excluded from primary claims.
  • External validity: Generalization is unknown beyond one synthetic enterprise domain, model/provider configuration, two languages, PostgreSQL, and a controlled semantic catalog.The study uses no real customer data, leaving applicability to other schemas, organizations, model families, providers, languages, and production traffic unresolved.
  • Statistical conclusion validity: Finite subgroup sizes, correlated template families, and multiple exploratory breakdowns constrain statistical interpretation despite case-aligned pairing and multiplicity correction.Language, class, and difficulty analyses should not become new primary hypotheses after observing the data.
  • Researcher validity: A single researcher designed the project and served as primary gold reviewer, so automated execution and reproducibility artifacts mitigate but do not remove author-in-the-loop bias.Independent replication and secondary annotation would strengthen future versions.

13 Limitations · 14 Reproducibility and Ethics

The study’s limitations include synthetic-data scope, restricted architectural and model comparisons, and no production-interface or deployment-safety certification. Reproducibility materials are broadly prepared for release, while raw responses remain subject to policy review and human responsibility governs scientific decisions.

  • 13 Limitations: SemPlan uses a synthetic domain that exercises enterprise-analytics semantics but cannot reproduce the full organizational context of real data systems.All benchmark data are synthetic and independently created.
  • 13 Limitations: The study does not fine-tune models, compare multiple model families, evaluate production user interfaces, or certify deployment safety.It also does not claim that the four architectures exhaust the design space.
  • 13 Limitations: Hybrid systems combining A3-style semantic requests with stronger policy enforcement or alternative state models remain future experiments.Such variants should be evaluated as new, pre-specified experiments rather than retrofitted into the completed F7 run.
  • 13 Limitations: Latency is not used as a comparative performance claim because of the reported instrumentation issue.This limitation concerns comparative interpretation rather than the release of other derived artifacts.
  • 14 Reproducibility and Ethics: Raw provider responses are hash-recorded locally, with public release deferred pending final policy and licensing review.Derived score records, frozen tables, figures, benchmark data, prompts, schemas, and deterministic reproduction paths can be released independently.
  • 14 Reproducibility and Ethics: The release candidate includes schemas, catalog files, frozen prompts, deterministic generators, manifests, result tables, figures, audit documents, and checksums.A no-key validation path can regenerate deterministic data and derived paper artifacts without additional paid provider calls.
  • 14 Reproducibility and Ethics: Code is licensed under Apache-2.0, while benchmark and dataset documentation is licensed under CC BY 4.0 unless otherwise stated.These licensing terms accompany the reproducibility release materials.
  • 14 Reproducibility and Ethics: Generative AI use in software development is disclosed, no AI system is an author, and human authors retain responsibility for scientific decisions and publication.Human responsibility explicitly covers design, validation, interpretation, claims, and publication decisions.

15 Conclusion · A Frozen Experiment Identifiers · B Frozen Prompt Hashes

SemPlan compares four LLM–deterministic-software architectures under a frozen bilingual enterprise-data benchmark, revealing a trade-off rather than a universal ranking. The conclusion also fixes the experiment’s immutable identifiers and architecture-specific prompt hashes for reproducibility.

  • 15 Conclusion: A4 achieved the lowest mean API cost and false-refusal rate, showing that efficiency and refusal behavior did not align with the top correctness result.The benchmark context reports A4 as the lowest-cost architecture and the one with the lowest false-refusal rate, while A3 had the highest correctness.
  • A Frozen Experiment Identifiers: Table 9 records the key immutable identifiers for the reported experiment.These identifiers are presented as frozen artifacts accompanying the benchmark results.
  • B Frozen Prompt Hashes: A1’s frozen prompt hash is 257c15c136c0a32108e41731a5d8b50c62c96b16e1ccdc2b973df92c6fad27d7.The hash is listed in the frozen prompt-hash record for A1.
  • B Frozen Prompt Hashes: A2’s frozen prompt hash is a50283334d378454192bd40d17a0747a225d3971a6203d8c9cc4443017293a05.The hash is listed in the frozen prompt-hash record for A2.
  • B Frozen Prompt Hashes: A3’s frozen prompt hash is 7a5cb95ccd3efb42feb4bedf2177250cdb2ee866758e8fbf93e90c85abb8fc95, and A4’s is 582d25398d5311e22b64d6472edb4e092dc85b095cc214acf86a1c37431cab51.Both hashes are listed in the same frozen prompt-hash record.
Loading 2608.13612v1…