PaperlayerView exact revision on arXiv

Source-linked AI summary

SemPlan: Benchmarking Structured Semantic Planning for LLM-Based Queries over Enterprise Data

Bruno Santos Teixeira

arXiv:2608.13612v1cs.AIcs.SE

TL;DR

Enterprise-data requests can be syntactically valid yet semantically wrong, policy-inconsistent, or operationally invalid. SemPlan compares four architectures in a controlled bilingual synthetic benchmark and finds trade-offs: A3 leads in correctness and repeatability, A4 in cost and false-refusal rate, and A1 in policy-correct behavior.

  • Problem

    SemPlan asks how structural constraints in LLM-mediated enterprise-data querying trade off answer correctness, policy behavior, failure modes, and cost.

    Source: §1 · p. 1

  • Method

    The paper compares four architecture patterns under a common experimental configuration using a bilingual synthetic benchmark for enterprise-data queries.

    Source: §1 · p. 2, §4.2 · p. 3

  • Results

    A3 had the highest observed answer correctness and repeatability, A4 the lowest mean API cost and false-refusal rate, and A1 the strongest policy-correct behavior and lowest unsafe-or-invalid rate.

    Source: §15 · p. 10

  • Takeaways & Limitations

    The findings support evaluating architectural trade-offs across multiple dimensions rather than ranking systems by a single accuracy number.

    Source: §15 · p. 10

    Generalization beyond one synthetic enterprise domain, one model/provider configuration, two languages, PostgreSQL, and a controlled semantic catalog is unknown.

    Source: §12 · p. 9

Abstract

from arXiv · show

Natural-language interfaces to enterprise data must translate underspecified requests into governed, executable behavior while controlling invalid queries, policy failures, cost, and nondeterminism. SemPlan Benchmark evaluates this architectural design space with a deterministic synthetic bilingual benchmark containing 1,800 cases in English and Brazilian Portuguese; 1,200 cases form the frozen scientific evaluation subset. Four architectures are compared under the same model configuration: direct SQL generation (A1), a bounded tool-agent baseline (A2), structured semantic-request generation followed by deterministic planning and execution (A3), and a clarification/stateful semantic-plan variant (A4). Across 4,800 primary records, answer correctness was low in absolute terms: 22.25% for A1, 22.58% for A2, 25.67% for A3, and 24.25% for A4. A3 had the highest observed correctness and significantly exceeded A1, A2, and A4 in the pre-specified paired correctness analysis, while A1 retained the highest policy-correct rate and the lowest unsafe-or-invalid rate. A4 had the lowest mean API cost and lowest false-refusal rate. On a preselected 150-case stability subset, answer-correct repeatability ranged from 92.00% to 98.67%. The results support a trade-off interpretation rather than a universal ranking: additional structural constraints changed failure modes and efficiency, but did not monotonically improve correctness or solve ambiguity and multi-turn state consistency.

1 Introduction

Enterprise natural-language data access must resolve business semantics, policy boundaries, and conversational context, not merely generate valid SQL. SemPlan compares structural constraint designs under fixed conditions to study their reliability trade-offs.

1 Introduction

  • Enterprise questions can be syntactically valid yet semantically wrong, policy-inconsistent, or operationally invalid because they involve business metrics, implicit dimensions, temporal scopes, authorization, unsupported actions, and conversational context.Ambiguity, domain modeling, and portability have also persisted as concerns in classical natural-language database interfaces.Source: §1 · p. 1
  • SemPlan asks how different forms and degrees of structural constraint trade off answer correctness, policy behavior, failure modes, cost, ambiguity, multi-turn state consistency, and repeatability.The study holds the model, benchmark cases, database snapshot, and evaluation rules fixed.Source: §1 · pp. 1–2
  • The study uses an independently generated synthetic enterprise domain, excludes customer and proprietary data, and is deliberately narrower than a claim of production readiness.Source: §1 · p. 2
  • SemPlan introduces a bilingual synthetic benchmark and compares four architecture patterns under a common experimental configuration rather than comparing different model families.Source: §1 · p. 2
  • The study reports paired evidence on answer correctness, policy behavior, typed failure outcomes, cost, ambiguity, multi-turn state consistency, and repeatability.Source: §1 · p. 2
  • The reproducibility package includes frozen prompts, manifests, raw-output hashes, score records, generated tables and figures, and no-key validation paths.Source: §1 · p. 2

2 Related Work · 3 Research Questions and Pre-Specified Hypotheses

SemPlan situates its evaluation among prior database-interface, text-to-SQL, tool-use, and reproducibility work, then pre-specifies five questions and directional hypotheses for comparing four architectures.

2 Related Work

3 Research Questions and Pre-Specified Hypotheses

  • Five research questions compare A1–A4 on correctness, failures and policy outcomes, cost and latency, ambiguity handling, and PATCH/REPLACE consistency.The study was frozen around these questions.Source: §3 · p. 2
  • Pre-evaluation hypotheses predicted semantic plans would improve correctness and reduce invalid or policy-violating execution, while A3 would cost no more than the tool agent and A4 would improve ambiguity and multi-turn consistency.The analysis preserves mixed and negative findings rather than redefining hypotheses after observing results.Source: §3 · p. 2

4 SemPlan Benchmark · 5 Evaluated Approaches

SemPlan uses deterministically generated bilingual cases over a governed synthetic commerce database, with 1,200 cases frozen for scientific evaluation. It compares four approaches that assign SQL generation, typed tools, semantic requests, and structured state and clarification to models and deterministic software in different ways.

4.1 Synthetic enterprise domain

  • Cases draw on deterministic Northstar Commerce data and parameterized semantic templates, with bilingual surface forms, validated identifiers, and normalized gold answers.The reference environment uses governed PostgreSQL views and read-only execution; cases are deduplicated by structural and lexical lineage before partitions are frozen.Source: §4.2 · p. 3, §4.1 · p. 3

4.2 Case generation, review, and splits

  • 1,800 cases comprise 900 en-US and 900 pt-BR cases, split across development (300), validation (300), public test (500), hidden test (300), multi-turn (200), and adversarial (200).Source: §4.2 · p. 3
  • The frozen scientific subset contains 1,200 public-test, hidden-test, multi-turn, and adversarial cases; development and validation are excluded from primary comparison.A single human reviewer approved 3,600 case/gold review records, a validity limitation the paper makes explicit.Source: §4.2 · p. 3

4.3 Illustrative bilingual case

  • Cases cover analytics, ambiguity, out-of-scope, multi-turn, and adversarial requests; a bilingual example maps equivalent revenue queries to one canonical semantic request.Difficulty labels use documented structural factors rather than model performance; primary evaluation uses frozen case artifacts, not manually rewritten examples.Source: §4.2 · p. 3, §4.3 · p. 3

5 Evaluated Approaches

  • Figure 1 distinguishes four designs: A1 permits model-authored SQL, A2 uses typed tools, and A3/A4 use typed semantic requests with deterministic execution; A4 adds state and clarification.All approaches use the same configured model, gpt-5.6-luna, reasoning effort low, and a 1,200-token output ceiling.Source: Figure 1 · p. 4, §5 · p. 4
  • A1 returns SQL or a typed refusal, then subjects generated SQL to parsing, allowlists, complexity limits, read-only authorization, timeouts, and row limits.The model receives the utterance, locale, reference date, governed schema, and metric definitions, but no gold SQL, plans, or answers.Source: §5 · p. 4
  • A2 composes a bounded number of schema-constrained analytics-tool calls; deterministic software implements operations, while arbitrary SQL and invented tools are disallowed.Source: §5 · p. 4
  • A3 emits a typed semantic request for deterministic validation and planning, while A4 extends that interface with structured state, typed clarifications, and PATCH/REPLACE rules.A3's application generates parameterized SQL or equivalent deterministic calls; A4 preserves state on out-of-scope turns unless reset, and execution remains deterministic after normalization.Source: §5 · p. 4

6 Experimental Setup

The experiment uses a frozen, case-aligned design across four approaches, with separate repeatability runs and pre-specified paired analyses. Latency is excluded from primary conclusions because captured typed failures make the resulting medians incomparable.

6 Experimental Setup

  • Each approach ran once on the same 1,200 scientific cases, database snapshot, model configuration, and frozen prompts, yielding 4,800 primary records.The frozen experiment used benchmark version 1.0.0-rc.2.Source: §6 · p. 4
  • A subset of 150 cases is executed two additional times per approach, producing 1,200 stability records; repeats assess only within-approach repeatability and do not inflate primary sample size.The complete scientific run contains 6,000 planned request identities.Source: §6 · p. 5
  • Binary endpoints use case-aligned McNemar tests and paired risk differences with 95% confidence intervals, with Holm-Bonferroni adjustment for the pre-specified comparison family.Skewed continuous outcomes such as API cost use paired bootstrap contrasts; failure types are distinguished before interpretation, while classifiable model-quality failures remain part of the scientific outcome.Source: §6 · p. 5
  • Latency is excluded from primary conclusions because captured typed failures record zero latency and are unevenly distributed, making medians incomparable as end-to-end measures.Token usage, provider cost, and failure counts remain reportable because they are independently recorded in provider usage and budget/result ledgers.Source: §6 · p. 5

7 Results

A3 had the highest observed answer correctness, but rates were modest and policy outcomes favored A1 on policy correctness and unsafe-or-invalid rate. Correctness was higher in en-US than pt-BR for every approach, descriptively.

7.1 Primary correctness and policy outcomes

  • 43.67% policy correctness and 31.00% unsafe-or-invalid for A1 were the best rates; A4 had the lowest false-refusal rate (0.17%) but the highest unsafe-or-invalid rate (64.08%).A3 had 37.33% policy correctness and 60.58% unsafe-or-invalid; paired analysis found A3 and A4 both below A1 on policy correctness.Source: §7.1 · p. 5

7.2 Exploratory language subgroup

  • 31.00% versus 20.33% correctness for A3 and 29.33% versus 19.17% for A4 in en-US versus pt-BR; subgroup findings were descriptive, not causal evidence.Correctness was consistently higher in en-US than pt-BR for all four approaches; correlated template families and the exploratory nature of the subgroup analysis limit interpretation.Source: §7.2 · p. 5

7.1 Primary correctness and policy outcomes

  • 25.67% for A3, 24.25% for A4, 22.58% for A2, and 22.25% for A1; A3 significantly exceeded each other approach in paired answer correctness.A3 exceeded A1 by 3.42 percentage points and A2 by 3.08 points; A4 was 1.42 points below A3. All three contrasts were Holm-adjusted significant.Source: §7.1 · p. 5, Figure 2 · p. 6, Table 3 · p. 6

8 Failure Analysis

Typed model-quality failures are scientific outcomes, not missing data; approaches shifted between ERROR outcomes and false refusals rather than simply removing errors.

8 Failure Analysis

  • Across 6,000 scientific records, EXECUTION_FAILED (1,762) and OUTPUT_SCHEMA_INVALID (1,051) were most frequent, followed by POLICY_VIOLATION (368), CATALOG_UNKNOWN_ID (78), and CFG_INVALID (24).Among the 4,800 primary records, corresponding counts were 1,402, 845, 287, 65, and 16, respectively.Source: §8 · p. 6, Table 5 · p. 7
  • EXECUTION_FAILED marks governed-execution rule failures; OUTPUT_SCHEMA_INVALID marks strict-contract failures; POLICY_VIOLATION marks guard rejection; CATALOG_UNKNOWN_ID marks unknown semantic identifiers; and CFG_INVALID marks configuration or contract invalidity.Source: §8 · p. 6
  • A1 produced 372 primary ERROR outcomes, versus 747 for A2, 727 for A3, and 769 for A4, but had 257 false refusals versus 22, 19, and 2 for A2–A4.Aggregate artifacts do not establish a unique causal mechanism for each error-code difference without case-level analysis.Source: §8 · p. 6

9 Stability, Ambiguity, and Multi-Turn Behavior · 10 Cost and Operational Results

Answer correctness was highly repeatable on the selected stability subset, but clarification and multi-turn state performance remained weak. A4 had the lowest observed provider-API cost per primary record, excluding total cost of ownership.

9 Stability, Ambiguity, and Multi-Turn Behavior

  • 98.67% repeatability for A3 was highest on the 150-case subset, followed by 98.00% for both A1 and A4 and 92.00% for A2.Repeatability measures stable answer correctness across the primary execution and two additional independent executions; it does not establish full determinism.Source: §9 · p. 7
  • 27.38% clarification-decision correctness for A3 was highest, versus 26.19% for A2, 8.33% for A4, and 0.00% for A1 across 84 ambiguity cases per approach.The clarification-enabled A4 design did not dominate the alternatives as pre-specified.Source: §9 · p. 7
  • 20.00% multi-turn sequence-state correctness for A1 exceeded 17.00% for A2, 14.00% for A3, and 14.50% for A4 across 200 cases.Structured state did not improve sequence-state correctness in this configuration.Source: §9 · p. 7

10 Cost and Operational Results

  • USD 0.000469 mean cost per primary record for A4 was lowest, compared with USD 0.000512 for A3, USD 0.000918 for A1, and USD 0.000927 for A2.The complete F7 scientific run recorded USD 4.028973 in provider-API charges; these observations exclude infrastructure and engineering costs and depend on the frozen model and price table.Source: §10 · p. 7
  • -USD 0.000415 was the paired mean cost difference for A3 versus A2, while the difference was -USD 0.000459 for A4 versus A2.A3 was also substantially cheaper than A1 and A2 under the frozen model and price table.Source: §10 · p. 7
  • 8,120,408 input tokens, 1,179,317 cached input tokens, 1,540,985 output tokens, and 639,950 reasoning tokens were recorded for provider usage.The counts support reproducibility and budget analysis, but should not be generalized to other models or future pricing; the full run recorded USD 4.028973 in API charges.Source: §10 · p. 7

11 Discussion

The discussion frames the architectures as trade-offs rather than a monotonic benefit from added structure: A3 leads on correctness and repeatability, while A1 and A4 have different policy and cost advantages. All approaches remain less accurate on pt-BR than en-US, and even the best is correct on only roughly one quarter of primary cases.

11 Discussion

  • A3 had the highest answer correctness and stability repeatability, with mean cost below A1/A2, while A4 had the lowest mean API cost but lower correctness and the highest unsafe-or-invalid rate.A4 was significantly below A3 on answer correctness; A3’s unsafe-or-invalid rate remained high.Source: §11 · pp. 7–8
  • A1 had the best policy-correct rate and lowest unsafe-or-invalid rate, whereas A2 showed only a small raw correctness increase over A1 and the weakest repeatability.The discussion also characterizes A2 as having a large error surface.Source: §11 · pp. 7–8
  • Direct SQL, semantic-request generation, and added clarification/state are proposed as possible mechanisms for the observed trade-offs, but these remain hypotheses rather than causal conclusions.The discussion suggests semantic requests narrow the model’s task, while added state and clarification may create useful governance capabilities and contract incompatibilities.Source: §11 · p. 8
  • Every approach was less accurate on pt-BR than en-US in the balanced frozen primary set, motivating independent multilingual review and additional model families.The discussion notes possible surface-form, lexical-mapping, model-behavior, or interaction effects and cautions against post-hoc retuning of the frozen benchmark.Source: §11 · p. 8
  • Even the best architecture was correct on roughly one quarter of primary cases across the benchmark’s clean, adversarial, multi-turn, and policy-sensitive distribution.The discussion presents SemPlan as evidence about comparative architectural trade-offs, not certification that any tested design is deployment-ready.Source: §11 · p. 8

12 Threats to Validity

The study’s findings are bounded by implementation and measurement choices, a controlled synthetic setting, statistical constraints, and researcher involvement. Reproducibility measures mitigate but do not eliminate these threats.

Internal validity

  • Shared models, database snapshots, cases, and execution limits reduce but do not eliminate implementation asymmetry; prompt design and recovery behavior may affect comparisons.The F7 hotfix changed failure capture, converting database exceptions into typed outcomes without changing benchmark semantics.Source: §12 · p. 8

Construct validity

  • Correctness measures canonical-result equivalence, not full user usefulness or trust; three-run binary repeatability is not deterministic generation, and latency is excluded from primary claims.Policy correctness and unsafe-or-invalid rates cover only specified benchmark outcomes; latency instrumentation assigns zero to captured typed failures.Source: §12 · pp. 8–9

External validity

  • Generalization beyond one synthetic domain, model/provider configuration, two languages, PostgreSQL, and a controlled catalog is unknown; no real customer data were used.Transfer to other schemas, organizations, model families, providers, languages, or production traffic remains untested.Source: §12 · p. 9

Statistical conclusion validity

  • Case-aligned pairing and multiplicity correction strengthen primary comparisons, but finite subgroups, correlated templates, and exploratory breakdowns constrain interpretation.Language, class, and difficulty analyses should not become new primary hypotheses after observing the data.Source: §12 · p. 9

Researcher validity

  • A single researcher designed the project and served as primary gold reviewer, so reproducibility safeguards mitigate but do not remove author-in-the-loop bias.Independent replication and secondary annotation would strengthen future versions.Source: §12 · p. 9

13 Limitations · 14 Reproducibility and Ethics · 15 Conclusion

SemPlan’s results show architectural trade-offs rather than a universal winner: A3 had the strongest correctness results, while A4 and A1 led on different efficiency and policy measures. The study is limited in scope, but provides reproducible artifacts and identifies directions for future evaluation.

13 Limitations

  • The study does not fine-tune models, compare model families, evaluate production interfaces, certify deployment safety, or claim its four architectures exhaust the design space.Its synthetic enterprise-analytics domain cannot reproduce the full organizational context of real data systems.Source: §13 · p. 9
  • Hybrid variants combining A3-style semantic requests with stronger policy enforcement or alternative state models should be evaluated as new, pre-specified experiments, not retrofitted into the completed F7 run.Source: §13 · p. 9
  • Raw provider responses are hash-recorded locally, but public release awaits final policy and licensing review; derived records and reproducibility artifacts can be released independently.The paper avoids latency as a comparative performance claim because of an instrumentation issue.Source: §13 · p. 10

14 Reproducibility and Ethics

  • The release candidate includes schemas, catalog files, frozen prompts, deterministic generators, benchmark manifests, result tables, figures, audit documents, and checksums; a no-key path can regenerate data and derived artifacts without paid provider calls.Synthetic benchmark data are independently created; code is licensed under Apache-2.0 and benchmark/dataset documentation under CC BY 4.0 unless otherwise stated. The frozen scientific-result manifest binds hashes for raw responses, records, figures, tables, and the cost ledger.Source: §14 · p. 10
  • The paper discloses generative AI use in software development, names no AI system as an author, and assigns scientific and publication responsibility to the human author.Source: §14 · p. 10

15 Conclusion

  • A3 had the highest observed primary answer correctness and repeatability; A4 had the lowest mean API cost and false-refusal rate; A1 had the strongest policy-correct behavior and lowest unsafe-or-invalid rate.The paired analysis supports a real but modest correctness advantage for A3; ambiguity, multi-turn, and policy results show that additional structure does not yield uniformly better behavior.Source: §15 · p. 10
  • Future work should replicate the benchmark across additional model families, independent reviewers, data domains, and languages, and test pre-specified variants targeting observed ambiguity, state, and contract-failure modes.Source: §15 · p. 10

A Frozen Experiment Identifiers · B Frozen Prompt Hashes

This block records key immutable experiment identifiers and frozen prompt hashes for architectures A1–A4.

A Frozen Experiment Identifiers

  • Table 9 identifies the key immutable identifiers for the reported experiment.The supplied passage gives the table’s scope but not its identifier values.Source: Table 9 · p. 10

B Frozen Prompt Hashes

Loading 2608.13612v1…