PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Effective and Reliable LLM Agents via Dynamic Ontology

Xiaohui Zhang, Zequn Sun, Chengyuan Yang, Yuanning Cui, Lingbing Guo, Wei Hu

arXiv:2608.22974v1cs.AI

TL;DR

LLM agents lack explicit task-level semantics and can use evidence incompletely during brittle multi-step decisions, while constructing usable ontologies is difficult to scale. OaK dynamically builds and refines a task-oriented schema, knowledge graph, and typed reasoning functions; across TravelPlanner, CRMArenaPro, and ToolQA, it improves agents and evidence grounding. The framework’s construction cost, dependence on an LLM extractor and judge, and need for a reliable evaluator remain important boundaries.

  • Problem

    LLM agents often leave domain relations implicit, causing incomplete evidence use and brittle multi-step decisions, while conventional ontology construction is costly and difficult to scale.

  • Method

    OaK dynamically constructs a task-oriented schema, schema-guided knowledge graph, and typed reasoning functions, then refines them with training examples, task scores, and execution feedback.

  • Results

    Across TravelPlanner, CRMArenaPro, and ToolQA, OaK achieves the best aggregate performance with two LLM backbones and improves evidence grounding for multi-step reasoning.

  • Takeaways & Limitations

    Making domain semantics and reasoning procedures explicit can improve both effectiveness and inspectability in LLM agents.

  • Takeaways & Limitations

    OaK incurs graph-instantiation costs, depends on an LLM-based extractor and judge, and requires a reliable task evaluator.

Abstract

from arXiv · show

Large language model (LLM) agents rely heavily on knowledge encoded in model parameters or presented as unstructured context. In domain-specific tasks, this leaves important semantic connections implicit. This often results in incomplete evidence use and brittle multi-step decisions. Ontologies offer a way to externalize domain concepts and relations as machine-interpretable structures, but constructing task-usable ontologies traditionally requires substantial effort from domain experts and is difficult to scale. Automatic construction is also challenging: an ontology that appears semantically plausible may not contain the relational structures needed for actual decision making. We present OaK, an ontology-as-a-kernel framework that dynamically constructs and refines task-oriented ontologies for LLM agents. Given task requirements and training data, OaK constructs an ontology and its knowledge graph, generates task-adaptation functions for graph reasoning, and uses judge feedback to iteratively refine both. By making relevant concepts and relations explicit, the ontology grounds knowledge retrieval and multi-step decision making. We evaluate OaK on TravelPlanner, CRMArenaPro, and ToolQA. Results show that OaK improves standard LLM agents, strengthens evidence grounding, and boosts the reliability of multi-step reasoning.

1 Introduction

OaK addresses reliability gaps in increasingly autonomous LLM agents by making task semantics, evidence, and executable reasoning explicit. It dynamically constructs and refines a task-oriented ontology, knowledge graph, and reasoning functions, improving agents across several benchmarks.

  • Motivation: Longer, more autonomous executions make intermediate retrieval and tool-call errors propagate, while final-answer accuracy alone cannot reveal evidence use or failure causes.The gap also includes hidden influences from memory and risks such as long-tail tool failures and prompt injection.
  • Motivation: Conventional ontologies describe domain concepts and relations but do not necessarily specify executable operations, parameters, or acceptance conditions for agents.This motivates a task-oriented representation between the LLM and its tools.
  • Approach: OaK dynamically constructs a task-conditioned schema, typed reasoning functions, and a corresponding knowledge graph, then refines them using training examples and downstream feedback.The resulting kernel is frozen before inference.
  • Approach: The schema defines available concepts, properties, and relations, while functions perform retrieval, filtering, traversal, projection, aggregation, and multi-step reasoning over graph evidence.During inference, a ReAct agent selects a function and binds typed arguments; construction feedback is then used to refine the schema and functions.
  • Results: Across TravelPlanner, CRMArenaPro, and ToolQA, OaK improves standard LLM agents and strengthens evidence grounding for multi-step reasoning.The evaluation covers datasets with different task settings and reasoning requirements.
  • Contributions: The framework combines a task-oriented schema, executable functions, a schema-guided evidence graph, automated construction, and judge-driven refinement.Ablations report the importance of function composition, the function module, and iterative refinement.

2 Related Work

Prior work improves agent control, feedback-based revision, and graph-based evidence access, but these approaches generally do not structure the task interface as reusable computations. OaK instead combines an explicit domain schema, typed reasoning functions, and a task-specific graph refined with downstream feedback.

  • LLM agents: ReAct, procedural memory, and workflow-oriented methods primarily improve how agents organize and reuse their control processes.OaK complements this work by structuring the task interface itself.
  • Reliable agents: Reliable-agent research evaluates whether intermediate decisions remain controlled, evidence-grounded, and traceable, not only whether the endpoint answer is correct.Self-Refine and Reflexion revise responses or future decisions using critique or execution outcomes.
  • Graph for agents: Graph-based approaches support structured knowledge acquisition, retrieval, and reasoning, but generally treat graphs as external knowledge layers rather than reusable task-level computations.OaK constructs typed reasoning functions and grounds their execution in a data-dependent knowledge graph.
  • Graph for agents: OaK jointly refines a task-specific schema and function catalog with downstream feedback before freezing the kernel for inference.Its dynamicity refers to constructing these components for the task at hand and instantiating the corresponding graph from task data.

3 OaK Framework

OaK builds a task-oriented ontology kernel that combines a verified schema, a schema-guided knowledge graph, and executable reasoning functions. It refines this kernel on training examples and feedback, then freezes it so a ReAct agent answers unseen queries through typed, graph-grounded tools.

  • 3.1 Overview: The kernel combines schema S, function set F, and a schema-guided graph so the agent can retrieve and reason over explicit domain concepts and relations.S bounds expressible concepts and relations, while F provides executable operations over graph evidence.
  • 3.1 Overview: OaK constructs the kernel in a training-data loop and freezes the resulting schema and functions for inference on unseen queries.The construction stage refines the kernel; inference makes no further edits.
  • 3.2 Ontology Construction Loop: Schema construction analyzes task requirements, drafts typed entities and relations, and formally verifies the schema before graph instantiation.Verification uses HermiT to check logical validity and returns failed drafts for repair.
  • 3.2 Ontology Construction Loop: Knowledge graphs are built with chunk–map–merge extraction, using schema-declared primary keys to canonicalize duplicate entities and relations.Chunking keeps extraction within the language model’s input budget, while merging reconnects relations to canonical nodes.
  • 3.2 Ontology Construction Loop: OaK composes generic graph operators into typed domain functions that package recurring multi-step reasoning into executable calls.The operator library includes lookup, traversal, projection, filtering, and aggregation; prior-round feedback guides function repair.
  • 3.3 Ontology-driven LLM Inference: During inference, the ReAct agent binds typed function arguments and executes them over the frozen graph, restricting computation to schema-constrained compositions.This routes answers through a verified semantic space intended to trace responses to grounded evidence.
  • 3.2 Ontology Construction Loop: A judge reviews task scores and execution trajectories, proposes add, delete, or modify repairs, and feeds them into subsequent construction rounds.The loop stops when no blocking fault remains or the iteration budget is exhausted.

4 Experiments

OaK is evaluated across multi-step planning, CRM execution, and compositional tool-use tasks, with ablations and robustness analyses examining its components and refinement process. Across these settings, OaK improves aggregate performance and evidence-grounded reasoning, while iterative refinement repairs schema and function deficiencies.

  • Tasks: The evaluation covers TravelPlanner, CRMArenaPro, and ToolQA, representing multi-day planning, CRM workflows, and compositional tool use over heterogeneous corpora.All methods use the official evaluation protocols and metrics, with two LLM backbones and a fixed ontology evaluator.
  • Main Results: OaK achieves the highest TravelPlanner final pass rate and leads macro-level metrics, indicating stronger jointly valid itinerary planning than isolated constraint satisfaction.Its schema-guided graph and adapted functions coordinate transportation, accommodation, dining, and budget dependencies across the complete itinerary.
  • Main Results: OaK records the highest CRMArenaPro average in both B2B and B2C settings and leads nearly all category comparisons, especially Policy and Database tasks.It explicitly represents CRM record types, fields, and relations, grounding cross-table queries, rule checks, and numerical calculations.
  • Main Results: OaK achieves the highest ToolQA weighted-average performance under both backbones and places first on most subset–backbone combinations.Typed procedures, explicit schemas, and grounded graphs connect lookup, filtering, traversal, and aggregation operations to appropriate records and relations.
  • Ablation Study: Removing any OaK component substantially reduces TravelPlanner performance, with the largest degradation from removing the function module or function composition.Without reusable composed procedures, the agent must reconstruct dependent operations for each query, weakening global plan consistency.
  • Ablation Study: The full OaK model outperforms all variants on CRMArenaPro and ToolQA, while removing function composition substantially harms cross-table and multi-step operation performance.On CRMArenaPro, composition removal causes the largest degradation; on ToolQA, graph access alone cannot replace schema-adapted procedures.
  • Iterative Refinement: Judge-guided refinement produces rapid early gains across all three benchmarks, with curves largely plateauing after rounds 4–5.The updates repair missing constraints, graph-mapping defects, and function-level gaps; one example adds hotel occupancy and minimum-stay constraints to schema and functions.
  • Efficiency and Robustness: OaK achieves the highest TravelPlanner final pass rate with fewer input tokens than ReAct and MemP, but incurs higher runtime and output-token usage than several baselines.The resource trade-off mainly reflects constructing a query-specific graph before planning.

5 Conclusion and Future Work

OaK turns domain semantics into an executable interface by combining a task-oriented schema, schema-guided graph, and typed reasoning functions. Across three benchmarks and two backbones, it achieves the best aggregate performance, while its costs and dependencies motivate future work.

  • OaK turns domain semantics into an executable interface by constructing a task-oriented schema, instantiating a schema-guided knowledge graph, and composing typed reasoning functions.
  • Across TravelPlanner, CRMArenaPro, and ToolQA, OaK achieves the best aggregate performance with two LLM backbones.
  • Ablations identify the function module, function composition, and iterative refinement as important components.
  • OaK still incurs graph-instantiation costs and depends on the LLM-based extractor, judge, and a reliable task evaluator.
  • Future work includes reusable and incrementally updated graphs, stronger verification or human feedback, and richer operator libraries for larger evolving domains.

A Schema Consistency Checks

The HermiT reasoner verifies five kinds of logical validity for the OWL ontology: disjointness, restrictions, properties, global satisfiability, and unsatisfiable classes.

  • Schema consistency checks: HermiT checks whether mutually exclusive categories are incorrectly forced to overlap, such as Restaurant being a subclass of City.
  • Schema consistency checks: It checks whether existential, universal, and cardinality restrictions jointly create contradictions.
  • Schema consistency checks: It verifies that declared property features remain compatible with the schema’s other axioms.
  • Schema consistency checks: It checks whether the schema is logically satisfiable as a whole rather than only in isolated fragments.
  • Schema consistency checks: It detects classes that cannot have an instance once type definitions, relations, and constraints are combined.

B Generic Operator Library

The generic operator library provides reusable graph operations for lookup, traversal, projection, filtering, runtime-slot extraction, and aggregation.

  • Operator library: The library exposes nine public operators, while underscore-prefixed helpers remain implementation details.
  • Operator library: Runtime-slot extraction maps natural-language queries to declared typed slots while retaining explicit or strongly implied constraints and their evidence.
  • Operator library: Entity lookup retrieves graph entities using types, identifiers, properties, exact values, or fuzzy name and text matching.
  • Operator library: Relation traversal expands a bounded neighborhood from seed entities along selected relation types and directions.
  • Operator library: Property projection converts selected entity properties into flat rows while preserving identifiers, types, and primary keys.
  • Operator library: Filtering operators support categorical, relation-connectivity, numeric, and set-overlap constraints over graph entities.
  • Operator library: Aggregation computes count, sum, minimum, maximum, or average statistics, optionally grouped by a field.

C Multi-Seed Robustness

OaK’s TravelPlanner gains remain stable across random seeds under the DeepSeek-v4-flash backbone, indicating that the reported gains are not driven by one favorable seed.

  • Multi-seed robustness: 55.90±2.13 final pass rate is reported across three OaK runs with different random seeds.All entries are percentages.
  • Multi-seed robustness: 58.60±2.38 on CS Macro and 61.57±1.04 on HC Macro show stable macro-level performance across seeds.

D Ablation Study on GPT-4o-mini

Under gpt-4o-mini, OaK outperforms its ablations across TravelPlanner, CRMArenaPro, and ToolQA, with substantial gaps over reduced variants.

  • Experimental setup: The ablation study reports the same three variants as the main-text study, using gpt-4o-mini as the backbone.The corresponding TravelPlanner, CRMArenaPro, and ToolQA results are shown in Figures 9–11.
  • TravelPlanner: TravelPlanner: OaK achieves the highest score on all five metrics under gpt-4o-mini.Its final pass rate falls from 19.70 to 9.90 without the function module, with degradation also visible on hard-constraint metrics.
  • CRMArenaPro: CRMArenaPro: OaK reaches 65.78 on average score, while the strongest ablation remains below 30 in both B2B and B2C settings.The full model clearly outperforms all variants in both settings.
  • ToolQA: ToolQA: OaK reaches 59.32 on weighted average and outperforms all variants on every subset.Ablations remain below 40, with the clearest gaps on Coffee, DBLP, and Yelp.
Loading 2608.22974v1…