PaperlayerView exact revision on arXiv

Source-linked AI summary

Symbolic Separation: Grounding Deep Agents in Knowledge Graphs for Trustworthy Operational Data Analytics

Baibek Davletiyarov, Junaid Ahmed Khan, Andrea Bartolini

arXiv:2609.17107v1cs.AI

TL;DR

Text-to-query and tool-using agents remain unreliable on heterogeneous operational data, where relations between sources are difficult to compose reliably. The paper separates free-form agent reasoning from data access through an ontology-constrained Virtual Knowledge Graph with pre-execution validation. On M100 ExaData, the approach improves end-to-end success over a non-symbolic configuration while preventing silent data-integrity errors and reducing token cost.

  • Problem

    Agents must compose relations across heterogeneous operational data, while existing text-to-query approaches remain unreliable on real-world and multi-step questions.

    Source: §I · p. 2

  • Method

    Symbolic separation lets a deep agent reason freely but routes data access through an ontology-constrained Virtual Knowledge Graph that validates queries before execution.

    Source: §I · p. 2

  • Results

    12/14 end-to-end tasks succeeded for the Neurosymbolic Deep Analyst on Qwen3.6-35B-A3B, versus 6/14 for the strongest non-symbolic configuration.

    Source: §V · p. 13

  • Takeaways & Limitations

    The approach prevented silent data-integrity errors that syntactic checks miss and used roughly 2.4× lower token cost on successful queries.

    Source: §V · p. 13

    The ontology’s expressiveness bounds both what the agent can answer and which hallucinations its validator can catch.

    Source: 1) An ontology-constrained knowledge… · p. 6

Abstract

from arXiv · show

Generative AI promises natural language access to the massive numerical telemetry of data centers and Industry 4.0 installations, yet text-to-query and tool-using agents stay unreliable: even frontier models answer little more than half of real-world database questions, and far fewer of the multi-step, operational ones, because the LLM must compose how heterogeneous sources relate and hallucinates the relations, not just the fields. We propose symbolic separation: a deep agent reasons freely but may act on data only through an ontology-constrained Virtual Knowledge Graph with deterministic pre-execution validation. Unlike a tool API's interface contract, this domain-semantic contract turns a complex question into one validated graph traversal instead of LLM-inferred joins. Instantiated as the Neurosymbolic Deep Analyst and evaluated on 49.9 TB of superconputer telemetry against a rigid workflow and a non-symbolic ablation, it raises end-to-end task success from 43% to 86%, prevents silent data-integrity errors that no syntactic check catches, and cuts token cost by 2.4x, letting a smaller on-premise model outperform a larger one.

I. INTRODUCTION

The introduction asks whether deep agents can move beyond EXASAGE’s rigid workflow and whether reliable operational-telemetry analysis requires symbolic data representation. It proposes symbolic separation and reports higher task success, retrieval accuracy, and efficiency for the resulting analyst.

I. INTRODUCTION

  • Text-to-database systems remain unreliable: GPT-4 reaches 40.08% execution accuracy on BIRD, while GPT-4o reaches 23.81% task-level pass@5 on dynamic multi-turn workloads.The introduction also reports that 6.91% of BIRD ground-truth queries contain undetected semantic errors.Source: §I · p. 1
  • Operational telemetry agents answer about 73% of stateless queries, but only about 34% of stateful and 10% of incident queries; schema confusion and source selection dominate failures.For IoT sensor-analysis code, import, datetime-format, and column-name errors together account for over 80% of failures.Source: §I · p. 1
  • EXASAGE previously reached 93.6% answer accuracy versus 25% for direct LLM-to-NoSQL, but its fixed query archetypes and single-shot workflow limit generalization and recovery.Its final validation catches execution syntax failures, not logically flawed queries that execute silently.Source: §I · p. 2
  • Symbolic separation lets the agent reason freely but permits data access only through an ontology-constrained graph that validates each access against the schema before execution.The domain-semantic contract makes a complex question a traversal over pre-existing, validated relationships rather than LLM-inferred joins.Source: §I · p. 2
  • The proposed analyst combines a deep-agent coordinator, a retrieval subagent, and a code agent; its conclusion favors focused symbolic subagents over adding an agentic loop to a monolithic prompt.The retrieval subagent uses an EXASAGE Reflect Agent for planning and retrieval, while the code agent performs statistical analysis and visualization in a sandbox.Source: §I · pp. 2–3
  • 86% end-to-end analytics success with Qwen3.6-35B-A3B, versus 7% for EXASAGE and 43% for a non-symbolic deep analyst.The comparison covers queries combining data retrieval and code execution.Source: §I · pp. 2–3
  • 88% accuracy on retrieval-only HPC operational-data queries, outperforming the state-of-the-art baselines.This result is reported separately from the end-to-end analytics comparison.Source: §I · p. 3
  • The EXASAGE Reflect Agent uses 2.4× fewer median tokens than the best alternative configuration, produces zero hallucinated outputs, and uses fail-fast retries.The stated retry behavior avoids wasteful computation.Source: §I · p. 3

II. STATE OF THE ART

The section situates operational data analytics in data centers and describes the scale and heterogeneity of the M100 ExaData corpus.

A. Operational Data Analytics for Data Centers

  • Data-center and HPC systems use operational data analytics frameworks to absorb heterogeneous, schema-less streams in NoSQL stores, leaving users to establish relationships between sources.These systems are instrumented with millions of sensors.Source: A. Operational Data Analytics for Data… · p. 3
  • M100 ExaData contains management, workload, facility, and infrastructure data from more than 980 compute nodes over two and a half years, totaling 49.9 TB uncompressed.The passage describes it as the largest public supercomputer dataset to date.Source: A. Operational Data Analytics for Data… · p. 3
  • The corpus is partitioned across nine Parquet plugins with distinct, sometimes inconsistent schemas, and has supported thermal-hazard prediction, anomaly detection, and predictive maintenance.Each analysis still demands deep query expertise.Source: A. Operational Data Analytics for Data… · p. 3

B. Knowledge Graphs for Heterogeneous Telemetry

Virtual Knowledge Graphs address telemetry heterogeneity without materializing the full graph, while existing LLM–KG approaches differ in how much relational composition they leave to the model. The paper frames symbolic separation as a domain-semantic contract that pre-connects relations and validates queries before execution.

B. Knowledge Graphs for Heterogeneous Telemetry

  • A full telemetry KG raises one-month, one-collector storage from 4.00 GiB to ∼3 TiB, while a VKG materializes only the query’s required subgraph.EXASAGE’s VKG reached a maximum storage size of 0.17 GiB across evaluated queries.Source: B. Knowledge Graphs for Heterogeneous… · p. 3

C. LLMs + Knowledge Graphs Taxonomy

  • KG-RAG and graph-guided reasoning leave final relational composition to the LLM, while text-to-SQL requires it to author joins.Grammar-constrained decoding checks lexical and syntactic form, but not schema validity or domain invariants.Source: C. LLMs + Knowledge Graphs Taxonomy · p. 3
  • Semantic-layer agents validate individual calls but leave cross-call relations to the LLM; telemetry agents particularly fail on multi-source composition and state.The cited telemetry evaluation reports ∼73% success on stateless, ∼34% on stateful, and ∼10% on incident queries.Source: C. LLMs + Knowledge Graphs Taxonomy · pp. 3–4
  • OBDA/VKG makes the ontology the query vocabulary and declares relationships once, but prior EXASAGE evaluation used 1K archetype-generated queries and was not an end-to-end analyst task.SPINACH’s reported failures were mainly property/relation misuse and insufficiently complex SPARQL composition, rather than surface formatting.Source: C. LLMs + Knowledge Graphs Taxonomy · p. 4
  • The paper positions prior telemetry-agent benchmarks as diagnostics rather than grounded systems and notes that earlier agentic work had not applied symbolic separation to timeseries or IoT data.Its related-work taxonomy distinguishes KG-augmented generation, semantic parsing, tool-use agents, and ontology-based access.Source: C. LLMs + Knowledge Graphs Taxonomy · pp. 3–4
  • The revised approach names symbolic separation: pre-connected ontology relations and deterministic validation replace LLM-inferred joins on complex questions.The validator rejects classes, properties, and domain-range violations before execution; the paper claims a trustworthiness-framed, deep-agent instantiation for operational analytics.Source: C. LLMs + Knowledge Graphs Taxonomy · p. 4

III. METHODOLOGY

Neurosymbolic Deep Analyst coordinates retrieval and analysis through specialist agents, with telemetry access confined to an ontology-constrained Virtual Knowledge Graph. The methodology combines interactive symbolic retrieval, sandboxed code execution, and traceable agent operations.

A. Architecture Overview

  • The coordinator plans and delegates, the EXASAGE Reflect Agent retrieves through an ontology-constrained per-query VKG, and the Deep Code Agent analyzes data in an isolated sandbox.The architecture is organized as containerized services communicating through MCP; the figure caption identifies it as the reference architecture.Source: A. Architecture Overview · pp. 4–5, Figure 1 · p. 5
  • A request is decomposed into ordered tasks: retrieval returns compact typed results, then the Deep Code Agent runs Python and returns analysis artifacts for presentation.Large retrieval outputs can be passed as CSV references, and interactions are recorded as nested traces.Source: A. Architecture Overview · pp. 4–5
  • The symbolic boundary prevents the retrieval LLM from accessing raw telemetry; every generated SPARQL query is checked against the ontology before execution.The A3 non-symbolic baseline removes this boundary and has the retrieval subagent read the raw data lake directly.Source: A. Architecture Overview · p. 5

B. Components

  • The coordinator is an orchestrator that decomposes requests, routes retrieval and analysis, and presents resulting artifacts; retries are bounded when a subagent reports an error.Its layered prompt includes shared workflow rules, configuration-specific routing, tool descriptions, and runtime conversation context.Source: B. Components · pp. 5–6
  • The EXASAGE Reflect Agent retrieves exclusively through the ontology-grounded VKG and supports query, clarification, and CSV-export tools.Its interactive, stateful pipeline keeps raw tables and fabricated column names out of the reasoning layer.Source: B. Components · p. 6
  • The Deep Code Agent runs analysis and visualization code in per-user sandboxes with no outbound network, gVisor isolation, and restricted input/output mounts.A static auditor flags medium- and high-risk imports before execution; the sandbox lifecycle includes initialization, upload, execution, artifact retrieval, and shutdown.Source: B. Components · p. 6
  • The Deep Code Agent turns retrieved telemetry into statistical analyses and visualizations, extending the system beyond question answering.The paper identifies what-if analysis as a future integration direction.Source: B. Components · p. 6
  • MetricsExpert supplies prompt-injected reference knowledge for resolving natural-language metric descriptions to exact metric names, while LangFuse traces record generations, tool calls, and dispatches.The reference covers M100 plugins and their metrics, units, sampling periods, and descriptions; traces support audit and token accounting.Source: B. Components · p. 6

C. The Symbolic Core: Ontology, Knowledge Graph, and the EXASAGE Reflect Agent

The EXASAGE Reflect Agent confines data access to a symbolic layer through pre-connected graph relations and deterministic ontology conformance checks. This design replaces query-time join inference with a validated graph traversal.

C. The Symbolic Core: Ontology, Knowledge Graph, and the EXASAGE Reflect Agent

  • VKG construction connects relevant telemetry as one RDF graph, so complex questions use an ontology-validated traversal rather than LLM-inferred joins.The cited relations include job–node, node–sensor–reading, and node–rack–room.Source: C. The Symbolic Core: Ontology,… · p. 6
  • A deterministic validator rejects queries naming ontology-absent classes or properties, or violating domain/range combinations, before execution.Source: C. The Symbolic Core: Ontology,… · p. 6
  • The paper presents pre-connected relations and conformance checking as the mechanisms distinguishing the EXASAGE Reflect Agent from the EXASAGE Query Tool.Source: C. The Symbolic Core: Ontology,… · p. 6

1) An ontology-constrained knowledge graph:

The revised ODA ontology expands the system model and separates physical, logical, and external entities. Its structure broadens expressible analytics while enabling ontology-conformance checks over class hierarchies.

1) An ontology-constrained knowledge graph:

  • The prior ontology modeled only a compute-node-centered hierarchy, lacked cooling and power equipment and sustainability metrics, and offered weak relationships for multi-hop reasoning.It also lacked rooms, generic containment, and a clear separation of physical and logical entities.Source: 1) An ontology-constrained knowledge… · pp. 6–7
  • The revised ontology has 308 axioms, 36 classes, 23 object properties, and 55 data properties, organized under PhysicalEntity, LogicalEntity, and ExternalEntity.The three top-level classes distinguish infrastructure, operational, and environmental concepts.Source: 1) An ontology-constrained knowledge… · p. 7
  • PhysicalEntity covers infrastructure and equipment, connected through transitive containment, with sensors attachable to any physical entity.Its scope includes power and cooling components such as UPS, PDU, Chiller, and CRAC.Source: 1) An ontology-constrained knowledge… · p. 7
  • LogicalEntity represents cluster, workload, job, metric, and time concepts independently of physical infrastructure.This layer supports reasoning about execution, performance, and operational state.Source: 1) An ontology-constrained knowledge… · p. 7
  • ExternalEntity includes weather and carbon-intensity signals, enabling environmental factors to be queried alongside internal system state.The revised ontology’s layered separation and transitive containment support hierarchy-aware validation, while its added equipment and sustainability entities expand expressible queries.Source: 1) An ontology-constrained knowledge… · p. 7

2) From EXASAGE Query Tool to the EXASAGE Reflect Agent: a redesigned, validated pipeline:

The EXASAGE Reflect Agent replaces the rigid query workflow with staged, interactive extraction and ontology-validated query generation. Deterministic checks and bounded correction govern access to query-specific virtual graphs and execution.

2) From EXASAGE Query Tool to the EXASAGE Reflect Agent: a redesigned, validated pipeline:

  • Input validation extracts fixed infrastructure entities with rules, while LLM-based time extraction normalizes colloquial expressions into ISO-8601 timestamps.Metric extraction is redesigned to ground abstract terms that may map to different metrics depending on context.Source: 2) From EXASAGE Query Tool to the… · p. 8
  • Multi-metric requests are decomposed into context-preserving sub-queries and grounded to plugin-specific metrics, with uncertain retrieval prompting clarification or failure.The pipeline combines LLM reasoning with embedding retrieval; an example query about GPU and CPU utilization is split and grounded to two metrics.Source: 2) From EXASAGE Query Tool to the… · pp. 8–9
  • For each request, the system builds a deterministic, query-specific named-graph VKG rather than materializing the full graph; graph-assembly errors abort the request.A cache reuses compatible named graphs and evicts entries by least recently used when capacity is exceeded.Source: 2) From EXASAGE Query Tool to the… · p. 9
  • Generated SPARQL is checked against ontology classes, properties, types, bindings, filters, and graph connectivity before execution.A bounded repair loop may adjust SELECT and GROUP BY clauses, while leaving WHERE patterns and filter logic immutable.Source: 2) From EXASAGE Query Tool to the… · pp. 9–10
  • The lifecycle pauses for clarification and returns to validation, but VKG construction errors fail directly while SPARQL generation can retry within a bounded loop.Successful requests proceed to query execution and then completion.Source: 2) From EXASAGE Query Tool to the… · p. 10

IV. EXPERIMENTAL RESULTS

The evaluation compares the proposed analyst, the EXASAGE Query Tool, and a non-symbolic analyst on end-to-end and retrieval-only query sets. Experiments use M100 ExaData and locally served open-weight models.

IV. EXPERIMENTAL RESULTS

  • The evaluation uses 14 newly proposed end-to-end analytics queries and 75 retrieval-only queries derived from recent HPC operational data analytics studies.Five HPC researchers rephrased the retrieval questions drafted from the studies' data pipelines.Source: §IV · pp. 10–11
  • Experiments use M100 ExaData and two locally served open-weight models, GPT-OSS-120B and Qwen3.6-35B-A3B.Runs use an NVIDIA H100 80 GB GPU, with LangFuse tracing for query execution graphs and token accounting.Source: §IV · p. 10
  • Three configurations share the coordinator, skills, LLM, and hardware, differing only in the backend used by the data retriever.The comparison includes the proposed EXASAGE Reflect Agent, the EXASAGE Query Tool, and a non-symbolic tool that queries the raw datalake.Source: §IV · p. 10

A. End-to-end data analytics task comparison

On 14 end-to-end tasks, the Neurosymbolic Deep Analyst with Qwen3.6-35B-A3B achieves 86% success, ahead of the non-symbolic configuration and EXASAGE Query Tool. It also uses fewer tokens than the best alternative and produces no hallucinated answers.

A. End-to-end data analytics task comparison

  • 86% success for the proposed Neurosymbolic Deep Analyst with Qwen3.6-35B-A3B compares with 29% for the non-symbolic Datalake Query Tool across 14 tasks.The proposed configuration records 12 correct, 1 incomplete, and 1 failed result.Source: Table I · p. 11, A. End-to-end data analytics task… · p. 12
  • 86% success for the proposed configuration contrasts with 7% for EXASAGE Query Tool using Qwen3.6-35B-A3B, while the tool scores 0% with GPT-OSS-120B.The two proposed-configuration failures reached the maximum context length.Source: A. End-to-end data analytics task… · p. 11
  • 43% success with GPT-OSS-120B but 14% with Qwen3.6-35B-A3B for the non-symbolic configuration shows that its result varies across the tested models.Only five of its six successful queries were correct.Source: A. End-to-end data analytics task… · p. 11
  • 300k median tokens per successful query for EXASAGE Reflect Agent with Qwen3.6-35B-A3B compares with 713k for Datalake Query Tool with GPT-OSS-120B, a 2.4× token efficiency improvement.The Reflect Agent has a median of 0 retries; the Datalake Query Tool has a median of 4 retries on failed queries.Source: A. End-to-end data analytics task… · pp. 11–12
  • The Reflect Agent rarely retries beyond one attempt, while the Datalake Query Tool can repeatedly retry failing queries without recovery.One Datalake Query Tool query had 14 retries and consumed millions of tokens.Source: A. End-to-end data analytics task… · p. 11
  • The Datalake Query Tool produces hallucinated outputs on Q11 and Q12, whereas the EXASAGE Reflect Agent produces zero hallucinated answers.The paper distinguishes plausible but factually wrong answers from hard failures.Source: A. End-to-end data analytics task… · p. 11

B. Data retrieval comparison

The rigid EXASAGE Query Tool outperforms the Datalake Query Tool on the 75-query retrieval study, but free-form phrasing exposes weaknesses in query generation and answer composition. The EXASAGE Reflect Agent substantially improves correct retrievals over the rigid tool.

B. Data retrieval comparison

  • 42/75 (56%) queries are answered by EXASAGE Query Tool, versus 36/75 (48%) for Datalake Query Tool.EXASAGE Query Tool performs better on templated queries but degrades on free-form phrasing.Source: B. Data retrieval comparison · p. 12, Table III · p. 12
  • 46.3% SPARQL-generation and 42.7% final-answer-composition accuracy trail 63.4% entity-extraction accuracy for EXASAGE Query Tool.The component breakdown locates the free-form phrasing loss in SPARQL generation and final-answer composition rather than entity extraction.Source: B. Data retrieval comparison · p. 12, Table IV · p. 12
  • 66/75 (88%) retrievals are correct for EXASAGE Reflect Agent, roughly double EXASAGE Query Tool’s result; residual failures involve query framing, SPARQL syntax, and out-of-context inputs.The comparison attributes the improvement to decomposing NL-to-SPARQL into specialized stages, not to agentic infrastructure alone.Source: B. Data retrieval comparison · p. 12

C. The Deep Code Agent (analysis sub-agent)

The Deep Code Agent processes operational telemetry into plots for distinct analytical tasks. Its successful outputs include time-series correlation, comparative profiling, and distribution analysis with outlier detection.

C. The Deep Code Agent (analysis sub-agent)

  • The evaluation presents three analytical tasks, Q4, Q9, and Q6, using EXASAGE Reflect Agent with Qwen3.6-35B-A3B for data processing and visualization.The reported outputs include inlet-temperature distribution, job power profiles, and node power and ambient-temperature trends.Source: C. The Deep Code Agent (analysis… · p. 12
  • The agent successfully generates plots requiring time-series correlation, comparative profiling, and distribution analysis with outlier detection.The authors characterize the analysis sub-agent as a cognitive extension of EXASAGE rather than merely a plotting tool.Source: C. The Deep Code Agent (analysis… · p. 12

V. CONCLUSION

The conclusion presents symbolic separation as the basis for a neurosymbolic deep agent that reasons freely but accesses data only through an ontology-constrained, prevalidated knowledge graph. In the reported evaluation, the approach improves end-to-end task success and avoids silent integrity errors.

V. CONCLUSION

  • Symbolic separation lets the agent reason freely but requires data access through an ontology-constrained knowledge graph that validates each access against the schema before execution.The approach is evaluated against two ablations on M100 ExaData.Source: §V · pp. 12–13
  • 12/14 end-to-end tasks succeed with the Neurosymbolic Deep Analyst, versus 6/14 for the strongest non-symbolic configuration on Qwen3.6-35B-A3B.The evaluation uses M100 ExaData and two open-weight LLMs.Source: §V · p. 13
  • The approach prevents silent data-integrity errors that syntactic checks do not catch and uses roughly 2.4× lower token cost on successful queries.The retrieval-level study attributes the gap with the published EXASAGE result to its single-shot formulation; the Reflect Agent roughly doubled correct retrievals.Source: §V · p. 13
  • Future work will add a visual validator, extend the ontology and VKG to stateful and incident queries, and evaluate other IoT and Industry 4.0 use cases and real systems.Source: §V · p. 13

APPENDIX A BENCHMARK QUERIES AND ANSWERS

Appendix A lists 14 end-to-end analytics queries, each requiring correct retrieval and a correct visualization or statistic. Appendix B presents system-generated outputs for temperature distribution, node time-series correlation, and job power profiling.

APPENDIX A BENCHMARK QUERIES AND ANSWERS

  • The 14 end-to-end queries Q1–Q14 require both correct retrieval and a correct visualization or statistic.The listed tasks span node counts, resource utilization, temperature, power, jobs, and weather data across specified time windows.Source: Appendix A · p. 15
  • The benchmark includes multi-variable analyses such as a heatmap correlation matrix and a comparison of top-10 users by job count and average power consumption.Source: Appendix A · p. 15

APPENDIX B DEEP CODE AGENT PERFORMANCE

  • 61,775 inlet-temperature readings fall in the 23–24°C peak range, while 361 outlier readings fall between 40–43°C; no readings occur from 31–39°C.The system-generated outputs also show a diurnal power–ambient-temperature pattern with a transient event around 06:30 UTC and compare power profiles for the three longest-running single-node jobs.Source: Appendix B · p. 15, Figure 1 · p. 15
Loading 2609.17107v1…