PaperlayerView exact revision on arXiv

Source-linked AI summary

GROUND: Reducing Hallucinations in LLM-Based Enterprise Analytics Through Governed Semantic Definitions

Aravind Sasidharan Pillai

arXiv:2608.26157v1cs.AI

TL;DR

Enterprise natural-language analytics needs governed business semantics because executable SQL can still use wrong metrics, grains, filters, joins, or access scope. GROUND supplies approved definitions and validates generated SQL against governance rules before execution; across the reported benchmarks, enforced properties held with zero violations, while judgment-dependent refusals remained fallible. The paper therefore positions semantic governance and enforcement as distinct requirements for reliable enterprise analytics.

  • Problem

    Enterprise reporting requires governed metrics, dimensions, joins, grain, filters, and row-level security beyond schema or documentation grounding.

  • Method

    GROUND retrieves approved semantic definitions and access policies, constrains SQL generation, and validates, retries, or abstains on governance violations.

  • Results

    Across the reported synthetic, real-data, and adversarial evaluations, GROUND's enforced properties held with zero violations, while semantic-only grounding still leaked data and some judgment-dependent behaviors remained fallible.

  • Takeaways & Limitations

    Semantic definitions improve metric correctness but do not guarantee governance; explicit policy injection and validation are required for row-level security enforcement.

  • Takeaways & Limitations

    The benchmark targets structured reporting, uses limited real-data and adversarial sets, and leaves production latency and token-cost effects unresolved.

Abstract

from arXiv · show

Natural-language analytics over enterprise data warehouses is increasingly important, but production use is limited by hallucinated metrics, invalid joins, wrong grain, unsafe data access, and unsupported explanations. Existing text-to-SQL systems often ground generation in database schemas or retrieved documentation, while enterprise reporting also requires governed business semantics: approved metrics, dimensions, join paths, filters, and row-level security. This paper introduces GROUND, Governed Retrieval Over Unified Normalized Definitions, a framework that constrains LLM-generated analytics to a governed semantic layer. GROUND supplies approved definitions, binds user intent to governed metrics and dimensions, and validates generated SQL against schema, metric, join, grain, filter, security, and cost rules before execution. On violations, it retries or abstains. In a 100-question synthetic enterprise-reporting benchmark, GROUND is compared with direct schema-only text-to-SQL, schema-RAG, and semantic-only grounding under one shared model. GROUND is the only system free of measured hallucinations across all six evaluated categories, while ungoverned systems violate row-level security on many questions. A semantic-only condition with exact metric definitions but no access policy still leaks data, showing that governance cannot be replaced by metric fidelity alone. The findings are replicated on real U.S. NHTSA vehicle-safety data with independent hand-authored gold and tested on an adversarial set across four models from three providers. GROUND's enforced guarantees, especially filters and row-level security, hold with zero violations on every model, while judgment-dependent behaviors such as refusing undefined metrics remain fallible.

Core contribution

GROUND treats enterprise analytics hallucination as a governance and semantics problem, not merely a SQL-generation problem. It combines governed semantic grounding with validation and policy enforcement, and evaluates these requirements through a dedicated taxonomy and benchmark.

  • GROUND combines governed semantic retrieval with validation, enforcing semantic definitions and access policies during SQL generation.
  • The framework evaluates schema, metric, join, grain, filter, security, cost, and explanation failures separately rather than relying only on execution accuracy.
  • Enterprise analytics requires approved metrics, reporting grain, sanctioned joins, required filters, row-level security, and organizational definitions beyond database schemas.
  • Semantic context alone leaves a residual governance gap: semantic-only grounding still violates row-level security despite perfect metric fidelity, whereas injected policy enforcement drives violations to zero.
  • GROUND's enforced properties are independent across models and runs, while recognizing undefined metrics and clarifying ambiguity remain capability-bound.
  • The benchmark compares direct schema-only generation, schema-RAG, semantic-only grounding, and full governed grounding under a shared model.

3 Problem Definition

The paper defines enterprise analytics correctness as a governed property of generated SQL and explanations, not merely successful execution. Its taxonomy identifies schema, metric, join, grain, filter, security, cost, and explanation failures, with abstention required for unsupported requests.

  • A trustworthy answer must satisfy schema, metric, grain, join, filter, security, and explanation-support requirements for the requesting user.
  • Schema hallucinations reference nonexistent tables or columns, such as dealer_profit_summary.net_revenue when neither object exists.
  • Metric and join failures invent business formulas or use invalid paths, such as defining customer lifetime value without a governed metric.
  • Security failures ignore row-level access constraints, and cost failures generate unnecessarily broad or expensive SQL.
  • Grain failures aggregate at the wrong level or double-count, while filter failures omit required semantic filters or include excluded work.
  • For ambiguous, undefined, or unauthorized requests, the system should clarify, reject, offer approved alternatives, or restrict results to permitted scope.

4 The GROUND Framework

GROUND routes natural-language questions through a governed semantic layer before execution. It retrieves and binds approved definitions, generates constrained SQL, validates governance rules, retries violations, and returns either validated SQL or rejection.

  • GROUND stands for Governed Retrieval Over Unified Normalized Definitions and forces analytics through a governed semantic layer before execution.
  • Question understanding classifies intent, metric, dimensions, time scope, and ambiguity before semantic retrieval.
  • Retrieval assembles approved metrics, dimensions, join paths, grain rules, required filters, and the user's row-level-security predicate.
  • Metric and grain binding maps user terms to governed definitions and expected aggregation grain, after which SQL uses approved semantic objects.
  • Validation checks schema, metric formula, joins, grain, filters, RLS, and cost; violations trigger feedback and up to three revisions.
  • The system returns auditable answers containing SQL, definitions, filters, warnings, or typed abstentions for unsupported requests.

5 Synthetic Enterprise Benchmark

The synthetic benchmark models governed automotive enterprise reporting with multiple data grains, fact tables, dealer hierarchies, time dimensions, and access policies. Its deterministic semantic-layer gold data supports evaluation of correctness and governance behavior across 100 questions.

  • The benchmark contains synthetic automotive reporting data with multiple grains, five fact tables, a dealer hierarchy, time dimensions, and access policies.
  • The semantic layer defines 10 approved metrics, each with a formula, base grain, required filters, and permitted dimension group.
  • It also specifies approved join edges, per-user RLS predicates for four synthetic users, and a business glossary for unsupported or ambiguous requests.
  • The benchmark contains 100 questions, with gold SQL for 70 additional questions generated deterministically from the semantic layer.
  • Answerable questions carry executable gold SQL and snapshots, while unsupported questions carry expected clarification, rejection, or security-restriction behaviors.

6 Experimental Setup

The experiment isolates governance context by comparing four systems under identical questions, database state, and model. GROUND adds governed semantic definitions, access policy, and validation/retry to the shared analytics task.

  • Systems and controls: Four systems are compared under the same questions and database state, with governance context as the only experimental difference.All systems share one model, isolating the effect of grounding.
  • Systems and controls: Direct LLM text-to-SQL uses raw table schemas and is expected to produce invented metrics, weak filters, and grain errors.
  • Systems and controls: Schema-RAG retrieves table and column descriptions, improving schema grounding but leaving business semantics weak.
  • Systems and controls: Semantic-only grounding supplies approved metric and dimension definitions but lacks validation, retry, and row-level-security enforcement.
  • Systems and controls: GROUND receives the full semantic layer, the user's RLS predicate, and validation/retry, trading higher overhead for lower hallucination risk.
  • Execution and protocol: The benchmark uses structured JSON outputs and executes queries against a committed SQLite database, while cloud-warehouse execution remains future work.DuckDB DDL and dialect-specific gold SQL are also provided for local warehouse-style execution.

7 Evaluation Metrics

The evaluation measures execution and result correctness alongside six hallucination categories, clarification, latency, token cost, and query cost. Results show that enforced governance eliminates measured filter and security violations, while judgment-dependent behaviors and deployment cost remain relevant boundaries.

  • Metrics: Execution accuracy measures successful SQL execution, while strict and value-based result accuracy assess exact outputs and metric correctness at the proper grain.
  • Metrics: Hallucination rates cover schema, metric, join, grain, filter, and security errors, with lower rates indicating better performance.
  • Metrics: Clarification accuracy measures correct refusals on abstention-expected questions; latency, token cost, and query-cost proxy capture operational overhead.
  • Synthetic results: GROUND achieves zero errors in all six measured hallucination categories, executes every emitted query at 1.00, and reaches 0.951 value-based result accuracy.Its strict accuracy is 0.793 because grouped outputs may omit identifier columns included in gold, whereas the value-based metric disregards that presentation difference.
  • Governance results: Semantic-only grounding reaches 0.00 filter hallucination but still violates row-level security on 35% of questions, while GROUND records 0.00 security violations.Direct LLM and Schema-RAG security violation rates are 0.78 and 0.53, respectively.
  • Clarification results: GROUND handles 94% of abstention traps correctly, whereas Schema-RAG over-refuses and reaches 0.646 execution accuracy.GROUND's one abstention miss still declined to generate SQL.
  • Operational trade-offs: GROUND's grounding packet costs approximately 10k prompt tokens, about 5× the direct baseline and 1.75× semantic-only, with latency around 1.3× direct.Only 9 of 82 SQL questions triggered validation revisions, averaging 0.15 retries per question.
  • Replication: On real NHTSA data, GROUND again records zero hallucinations and security violations, 100% execution, and correct values on all 40 questions.Semantic-only again leaks across manufacturer boundaries on 33% of questions despite exact metric and dimension definitions.

9 Error Analysis

GROUND records zero hallucinations, with remaining misses arising from presentation shape, under-specified questions, abstention-category mismatch, and data-value conventions rather than governance failures.

  • Residual errors: GROUND has zero hallucinations of any type; its five residual value or abstention misses are not governance failures.The misses involve presentation shape, omitted reporting years, and stricter abstention than the benchmark reference.
  • Residual errors: Three security cases enforced the correct row-level restrictions, but figures differed because questions omitted the reporting year.GROUND defaulted to year-to-date while the gold SQL assumed 2025; adding the explicit year corrected the benchmark-authoring gap.
  • Residual errors: GROUND returned correct comparison values in long format although the gold answer required wide format, triggering a value-based metric flag.Every number was present and correct; only row count and column names differed.
  • Residual errors: For “show margin by dealer,” GROUND rejected an undefined metric instead of asking for clarification, reducing clarification accuracy without hallucinating.The stricter abstention category differed from the reference behavior.
  • Error fixes: Adding a semantic-layer time-resolution convention corrected a quarter-label mismatch by resolving time scopes to date_value ranges.The earlier query compared Q4 2025 with a column storing 2025-Q4; the remaining benchmark gap therefore concerns data-value conventions and question or gold specification.
  • Baseline comparison: Baseline errors were dominated by omitted filters and security constraints, while schema-RAG also over-refused after partial schema retrieval.These failure modes motivate explicit governance and validation beyond retrieval alone.

10 Threats to Validity

The evaluation addresses synthetic-data and gold-generation concerns through real NHTSA replication, but remains limited by sample scale, single-run baselines, heuristic classifiers, and scope boundaries.

  • Data and evaluation: Synthetic evaluation and ungoverned baselines use one run per configuration, limiting precision of non-enforced error estimates.GROUND’s enforced governance metrics are deterministic at 0.000 ± 0.000 across three runs per model on both NHTSA sets.
  • Data and evaluation: The real-data and adversarial NHTSA evaluations contain 40 questions each, compared with 100 synthetic questions.The real-data replication uses independently hand-authored gold and reaches the same conclusions, partly addressing synthetic-data concerns.
  • Governance boundary: GROUND’s model-judgment behaviors remain fallible, including undefined-metric recognition and ambiguity handling.Only schema, metric formula, join, grain, filter, and row-level-security properties are enforced guarantees.
  • Measurement: Regex-plus-semantic hallucination classifiers have zero false positives in an oracle self-test but may miss violations in adversarial SQL.The classifier result therefore does not establish complete detection of every possible violation.
  • Scope: The benchmark targets structured reporting and excludes unstructured analytics and causal diagnosis.This scope boundary limits direct generalization beyond the evaluated reporting setting.

11 Discussion

The results support deploying enterprise LLM analytics as a governed data product whose semantic layer controls meaning, access, and auditability, with measurable overhead.

  • Governed deployment: The semantic layer functions as the control plane for business meaning, access, and auditability, especially row-level security.The semantic-only ablation shows that perfect metric definitions do not substitute for enforced access policy.
  • Governed deployment: GROUND returns the query, definitions used, and applied filters, supporting safer self-service analytics and auditable SQL generation.The framework also provides reusable metric governance and escalation paths for undefined KPIs.
  • Trade-offs: ~5× token usage and ~1.3× latency versus the direct baseline are the principal deployment costs, while validation retries average 0.15 per query.Prompt caching of the static semantic layer may recover much of the token cost in production.
  • Trade-offs: GROUND’s zero hallucinations and zero security violations contrast with 35–78% RLS violations for ungoverned systems.The authors identify this reliability difference as decisive for compliance-sensitive deployments, while caching and smaller-model sufficiency remain open questions.

12 Conclusion

GROUND combines governed semantic retrieval with SQL validation to reduce enterprise analytics hallucinations. Its enforced safety properties remain robust, while judgment-dependent refusal and clarification behaviors are fallible and the approach costs roughly 5× the direct baseline’s tokens.

  • Contribution: GROUND supplies approved metrics, dimensions, joins, grain rules, filters, and access policies before SQL generation, then validates queries with retries.The framework was evaluated on 100 synthetic questions and replicated on 40 real NHTSA questions with independent hand-authored gold.
  • Findings: GROUND achieves near-zero hallucinations and zero row-level-security violations, while ungoverned baselines violate RLS on 25–78% of questions.Ungoverned baselines also drop required filters on 50–73% of questions.
  • Implication: Safety-critical properties such as security, filters, and grain require validation rather than prompting alone.The adversarial test preserved enforced properties but exposed occasional fabricated metrics and imperfect ambiguity handling.
  • Limitation: GROUND uses roughly 5× the direct baseline’s token cost, although prompt caching of the static semantic layer may largely close the gap.This overhead is the main stated production trade-off.

13 Appendix A. Semantic-Layer YAML Excerpt

The appendix defines GROUND’s governed semantic-layer artifacts and shows how metrics, grain, filters, access rules, time resolution, and undefined metrics are encoded. It also specifies benchmark fields and an example rejection behavior for an undefined metric.

  • GROUND’s benchmark package separates metrics, dimensions, joins, row-level-security rules, and the business glossary.
  • Service Revenue is defined over fact_repair_order_line using customer-pay labor and parts amounts, with a required service_customer_pay filter set.
  • repair_order_count counts distinct orders even though rows originate from the line fact, enforcing an order-level grain guard.
  • The semantic layer encodes a West dealer predicate and requires time filtering through dim_calendar.date_value ranges rather than label equality.
  • customer_lifetime_value is explicitly undefined, so the example request to show it by dealer must be rejected rather than assigned an invented formula.
  • Benchmark records capture question identity, task type, test user, natural-language request, required metrics and dimensions, and time scope.

15 Appendix C. Reproducibility and Artifact Availability

The appendix documents the released artifacts, data provenance, execution environment, evaluation commands, model configurations, and reproducibility boundaries. It distinguishes deterministic governance checks from hosted-model judgments that can vary across runs and provider updates.

  • The complete benchmark, semantic layers, gold SQL, runners, evaluation harness, and committed outputs are released for regeneration and scoring.
  • The environment uses Python 3.9 with pandas, pyyaml, duckdb, and provider SDKs; scoring and self-tests do not require model API keys.
  • The synthetic data are canonical committed CSVs, while the NHTSA domain derives from public U.S. vehicle-safety files and contains 271,718 complaints.
  • The self-test scores gold SQL as an oracle across synthetic, NHTSA, and adversarial domains, while separate commands regenerate snapshots, run systems, and score variance.
  • Four models from three providers were evaluated: Claude Opus 4.8, Claude Sonnet 5, GPT-5.2, and Llama-3.3-70B.
  • For models rejecting strict structured output, the runner falls back to json_object and then plain-prompt JSON.
  • Only GROUND uses a validate-and-retry loop, with a default budget of 3 retries; synthetic evaluation and baselines are single-run, whereas NHTSA GROUND runs three times per model.
  • Governance metrics reproduce at 0.000 ± 0.000, but judgment-dependent metrics may drift across runs and hosted-model snapshot updates.
Loading 2608.26157v1…