Designing for Imperfect Prompts: A Simulation-Based Evaluation of an LLM-Based Enterprise Data Assistant Using Knowledge Graphs

Problem

• Business users often pose incomplete, unstructured prompts using domain-specific vocabulary, leading to inaccurate, incomplete, or inefficient data retrieval.Data sources may differ from user expectations in naming, scope, granularity, and terminology, while standalone LLMs cannot resolve all misalignment between intent, data semantics, and domain knowledge.Source: §1 · pp. 2–3

Research question

• The study addresses the gap in research on automatically enriching real imperfect user prompts with a semantic mediation layer, asking how LLM-based enterprise data assistants can improve them.The proposed mediator bridges user intent and source data by interpreting and contextualizing prompts.Source: §1 · p. 3

Design and setting

• In a second design cycle, the researchers evaluated schema-based mediation against knowledge-graph-based mediation using 158 employee interactions from a real-world CFO-unit deployment.The schema-based variant retrieves relevant database schemas from prompts; the knowledge-graph variant uses a graph automatically constructed from the database schema.Source: §1 · pp. 3–4

Results

• Knowledge-graph-based mediation outperformed schema-based mediation in Precision, Recall, F1-Score, prompt enrichment, and intent alignment in a simulation using 158 interactions from 29 employees.Intent completeness was assessed with an LLM-as-a-Judge and validated by two human annotators from the industry partner.Source: §1 · p. 4

Additional analysis

• Mediation effects depended on prompt content: schema retrieval aided prompts with complete schema information, broadened results correctly for incomplete prompts, and did not prevent errors from incorrect information.For prompts already containing all relevant schema information, additional retrieval was obligatory and made data retrieval more efficient.Source: §1 · p. 4

2.1 Conversational Data Assistants

• LLM-enabled assistants can reason over schemas and generate queries, yet domain jargon and gaps between analytical intent and operational formulation make user prompts semantically or pragmatically incomplete.Users may know what analysis they want but lack the vocabulary or schema awareness to express executable logic; prompt engineering, finetuning, and in-context learning are proposed domain-adaptation approaches.Source: §2.1 · p. 4
• Benchmarks progress from curated text-to-SQL queries toward ambiguity, conversational realism, and external knowledge, while the study’s enterprise prompts were shorthand and rarely reflected multiple valid interpretations.Spider and Spider 2.0 use carefully curated prompts; AmbiQT, AMBROSIA, and PRACTIQ introduce ambiguity or unanswerability, while BIRD-INTERACT emphasizes external knowledge and metadata. The study’s first design cycle found knowledge-driven mediation better suited to its observed prompt imperfections than treating them as multiple interpretations.Source: §2.1 · p. 5

2.2 Mediators and Knowledge Graphs

• Mediators encode and reuse domain knowledge to reconcile data sources; semantic mediation explicitly represents meaning to align user intentions with schemas across the semantic gap.Knowledge graphs model entities as nodes and semantic relationships as edges, and embeddings can support retrieval and semantic matching.Source: §2.2 · p. 5
• For relational databases, knowledge graphs can represent data entities or database semantics; entity representation captures more detail but is resource-intensive, while schema-semantics mapping is simpler but partial.The simpler mapping approach does not account for all relational constraints or integrity rules.Source: §2.2 · p. 5

Problem Identification and Motivation

• Users’ brief prompts often omitted schema references, used company-specific terminology, and made the assistant seem to restart contextual searches for repeated questions.The second design cycle follows a prior CFO-unit pilot; earlier engineering changes included richer metadata, task- and effort-based subagents, routing, vector-store examples, and prompt engineering.Source: §3 · pp. 6–7

Definition of Solution Objectives

• The proposed solution automatically builds a compact schema-level KG of tables, columns, and relationships, avoiding external domain knowledge and manual enrichment.It represents schema rather than individual data entities, enabling relationship traversal and joinpath inference while remaining updateable as the database changes.Source: §3 · p. 7

Design & Development

• The design compares KG-based mediation using an automatically generated database KG with schema-based mediation that retrieves schemas by semantic similarity.The comparison tests whether KG structure and reasoning offer a practical advantage in mediating user intent and database semantics.Source: §3 · p. 7

Evaluation

• Evaluation used 158 interactions from 29 users’ most recent 30 days of available logs, with word counts, LLM-based intent assessments, two human-annotation rounds, and two LLMs.The analysis also measured performance against a baseline to assess whether user intent was preserved when database information was exposed.Source: §3 · pp. 7–8
• The simulation findings indicate that KG-based mediation provides richer structural context and better intent alignment than schema-based mediation, while remaining efficient and maintainable.The study synthesizes these findings into a design principle for a knowledge-based semantic mediation layer.Source: §3 · p. 8

4.1 Problem Identification

• Prompts averaged 1.14 sentences and roughly 10 words, lacked schema references, and led the assistant to guess plausible but incorrect columns and joins.For example, it treated a VAT fiscal unity state attribute as a proxy for the concept instead of tracing subsidiary, parent, shareholding, and control relationships; fiscal unity could also mean income-tax or VAT unity.Source: §4.1 · p. 8
• Across 158 recent interactions, no prompt contained explicit schema references, and the short, single-intent prompts exposed the brittleness of hand-curated routing and few-shot examples.The configuration helped only in explicitly modeled domains with closely matching examples; unanticipated questions still produced incorrect answers, making exhaustive manual curation unsustainable at enterprise scale.Source: §4.1 · p. 8

4.2 Solution Objective

• The mediator’s requirements are to preserve user intent, ground prompts in domain and data knowledge, and operate automatically with low maintenance.The proposed knowledge graph captures tables, columns, semantic descriptions or metadata, and relationships such as foreign-key links to support interpretation and mapping to underlying data.Source: §4.2 · p. 9

5 Design & Development

• The design compares KG-based mediation, which uses graph structure and reasoning, with schema-based mediation, a lightweight baseline retrieving schemas by semantic similarity.Source: §5 · p. 9

5.1 Designing a Semantic Mediation Layer

• The mediation layer segments user intent, retrieves relevant schema or KG elements, and uses an LLM to refine the prompt against the source data.The KG variant additionally traverses connected tables and columns to capture direct and indirect relationships.Source: §5.1 · p. 9

5.2 Knowledge Graph Conceptualization

• The lightweight KG represents tables and columns with has_column and joins_on links, enabling direct, inferred, and multi-hop join reasoning.It contains 273 nodes and 540 edges; joinable relationships come from primary- and foreign-key constraints or heuristics. Nodes are embedded in ChromaDB for semantic retrieval, then graph traversal identifies join paths. The KG intentionally omits domain-specific metrics and fine-grained filters to isolate structural database knowledge.Source: §5.2 · p. 10

5.3 Demonstration

• In the audit-request example, KG mediation connected 21 nodes and 30 edges and identified the correct request-company path, while schema mediation produced the wrong audit-period query.The KG-based prompt links open requests to company name and ID; schema-based mediation identified 76 columns but selected incorrect ones.Source: Figure 5 · p. 11, §5.3 · p. 11

• In 158 interactions from 29 CFO-unit employees, evaluation combined LLM-as-a-Judge intent-completeness labels with two analysts’ TP, FP, and FN annotations.Analysts reviewed in two iterations, compared assessments, and resolved discrepancies before Precision, Recall, and F1 were calculated; the no-mediation baseline reported completeness only because it added no database elements.Source: §6 · p. 12
• KG-based mediation outperformed schema-based mediation across both LLMs in Precision, Recall, and F1, with slightly higher completeness.With GPT-4.1, KG achieved Precision 90.67% (Schema: 69.05%), Recall 88.31% (Schema: 73.11%), and F1-Score 89.47% (Schema: 71.02%). With Claude Sonnet 4, KG achieved Precision 87.23% (Schema: 75.17%), Recall 93.52% (Schema: 92.56%), and F1-Score 90.29% (Schema: 82.17%).Source: Table 1 · p. 12, §6 · p. 13
• For GPT-4.1, KG mediation achieved completeness of 93.04% versus 91.14% for the no-mediation baseline and 85.33% for schema mediation.Claude Sonnet 4 had similar completeness across conditions; the passage also notes that minimal prompt changes caused by missing domain or database knowledge are not captured by completeness.Source: §6 · p. 13
• In 30 end-to-end trials, correct mediated prompts sometimes avoided schema discovery, while erroneous assumed joins on complex schemas caused downstream query-planning and execution failures.KG performance depended on identifying the right entities and relations by semantically matching user phrasing to canonical KG nodes; non-database prompt checks found neither approach injected additional or incorrect knowledge. Schema retrieval averaged 1.1 seconds versus 2.7 seconds for KG retrieval.Source: §6 · p. 13

7 Discussion

• KG-based mediation better captures users’ specific intents and relationships than schema-based mediation, reducing missing or incorrect information and motivating a prompt-augmenting, context-aware mediation design principle.The proposed layer interprets user intentions and represents them as context-aware source data representations, reducing dependence on manual prompt engineering.Source: §7 · p. 14

7.1 Theoretical contribution

• Knowledge delivery matters alongside knowledge curation and retrieval: domain-specific source relationships mitigate imperfect prompts, while excessive or unconnected context can undermine LLM performance.Ambiguity depends on user intent, the represented domain, and the assistant’s ability to mediate between them; knowledge can extend capabilities beyond clarification and multi-query techniques.Source: §7.1 · p. 14

7.2 Practical implications

• Organizations should explicitly incorporate knowledge and tailor assistants to target users and domain characteristics, including users unfamiliar with SQL or underlying data structures.The study advises considering users’ roles and domain characteristics in addition to their experience with data-analysis tools and LLMs.Source: §7.2 · p. 15

7.3 Limitations and future research

• The study’s single-enterprise setting, 158 interactions from 29 users, and log-based simulation limit generalizability and leave real-time user effects untested.Future work could replicate across industries, functions, and larger prompt corpora, and use field deployments, controlled A/B studies, qualitative research, and user-centred end-to-end evaluations to assess longitudinal effects and evolving prompts.Source: §7.3 · p. 15
• Assistant performance depends on knowledge graph coverage and maintenance, and the study did not systematically vary graph density or test other graph configurations.Future research could develop governance, lifecycle practices, and stewardship tools; extend graphs with richer constraints and domain vocabulary; investigate hybrid or adaptive mediation; and test graph structures, retrieval strategies, subgraph granularity, and windowing trade-offs in latency and costs.Source: §7.3 · p. 15

8 Conclusion

• Semantic mediation combined with embedding models and knowledge graphs helps mitigate imperfect prompts and positively influences enterprise data assistant performance.Source: §8 · p. 15
• The study contributes a user-perspective design view by emphasizing the role of mediators and knowledge in LLM-based enterprise data assistants.Source: §8 · p. 15
• Enterprise assistants should externalize domain specifics, meet users where they are, and evolve as enterprise knowledge grows.Source: §8 · p. 15