PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoOntology: A Self-Evolving Ontology Layer for Data Agents

Meiduo Chong, Shaolei Zhang, Ju Fan, Xiaoyong Du

arXiv:2609.15779v1cs.AIcs.CLcs.DB

TL;DR

Data agents must understand heterogeneous data that sits outside them, but existing approaches struggle to scale or adapt to different agents. EvoOntology packages a self-evolving ontology as an MCP server that agents can query at runtime. Across benchmarks and backbones, it consistently outperforms strong baselines, with an average gain of +17.8 points over Baseline in DDR-Bench Trajectory-Wise accuracy.

  • Problem

    The agent–data gap remains: data agents must understand heterogeneous data outside themselves, while existing approaches are difficult to adapt to new data sources, tasks, and agents.

    Source: Introduction · pp. 1–2

  • Method

    EvoOntology packages a schema, content, and tool ontology as an MCP server and uses a builder agent with attribution-guided refinement and backbone-conditional paired evaluation.

    Source: Introduction · p. 2, Trajectory-Grounded Ontology Evolution · p. 4

  • Results

    EvoOntology consistently outperforms ReAct and traditional semantic-layer baselines across benchmarks and backbones; on DDR-Bench, it gains an average +17.8 points over Baseline in Trajectory-Wise accuracy.

    Source: Introduction · p. 2, Main Results · p. 4, Conclusion · p. 7

  • Takeaways & Limitations

    The authors conclude that EvoOntology offers an effective solution for helping data agents understand heterogeneous data.

    Source: Conclusion · p. 7

    A reported limitation was the missing reusable Content Layer description of a status field and its applicability condition.

    Source: Semantics · p. 11

Abstract

from arXiv · show

Data agents aim to fulfill natural-language instructions over heterogeneous data, including tables, files, and databases. However, data agents face a challenging agent-data gap: heterogeneous data resides outside the agent, while the agent can access it (e.g., column names and file paths) only through generic tools. Existing approaches either let agents directly explore raw data sources or inject manually constructed semantic layers into prompts. However, neither scales well to large heterogeneous data sources nor adapts to different agent behaviors. In this paper, we introduce EvoOntology, a self-evolving ontology layer for data agents. EvoOntology encapsulates the ontology as an MCP server comprising a schema layer, a content layer, and a tool layer, enabling agents to actively query and interact with the ontology at runtime. To this end, we introduce a builder agent for autonomous ontology construction and a self-evolution loop that continuously refines the ontology through attribution-guided typed edits that are accepted only after a backbone-conditional paired evaluation. Experiments on three well-adopted data-agent benchmarks with four LLM backbones demonstrate that EvoOntology consistently outperforms strong baselines and existing semantic-layer approaches, effectively bridging the agent-data gap and enabling more effective interaction with heterogeneous data. Code: https://github.com/ruc-datalab/EvoOntology

Introduction

Data agents face an agent–data gap because heterogeneous data is accessible only through generic tools, forcing blind exploration. EvoOntology addresses this with an interactive, self-evolving ontology layer that agents query through tools.

  • Because structure and content are not known a priori and agents see data only through generic tools, they must guess where concepts reside and inspect potentially irrelevant content.This mismatch is the agent–data gap that an intermediate ontology layer is intended to bridge.Source: Introduction · p. 1
  • Raw querying scales poorly to wide, heterogeneous sources, while semantic layers are impractical to inject wholesale and costly to build and adapt manually.These limitations motivate an effective, scalable ontology intermediate layer.Source: Introduction · pp. 1–2
  • EvoOntology encapsulates schema, content, and tool layers as an MCP server, letting agents query and manipulate an interactive ontology instead of passively consuming semantic metadata.The schema defines object types and reference rules, while the content layer stores domain knowledge and data mappings.Source: Introduction · p. 2
  • A builder agent grounds an initial ontology in probe-query observations, then attribution-guided refinements to schema, content, or tools are accepted only after paired evaluation on held-out data.The refinement loop uses agent interaction trajectories to identify ontology deficiencies.Source: Introduction · p. 2
  • Experiments on three data-agent benchmarks with four LLM backbones show that EvoOntology consistently and substantially outperforms strong baselines and existing semantic-layer approaches.The introduction reports performance across multiple benchmarks and backbones but provides no metric values here.Source: Introduction · p. 2

Related Work · Method

EvoOntology builds an evidence-grounded ontology around a data-agent workload, exposes it through runtime tools, and refines it using historical trajectories. Its updates are localized and retained only after backbone-conditional paired validation.

Related Work

Method

  • EvoOntology combines a builder that constructs an evidence-grounded initial state with an evolution agent that refines it from historical trajectories.Its versioned state comprises content, schema, and tool layers and is built around the workload, accessed through the agent’s tool interface, and adapted from execution history.Source: Method · p. 2

Agent-First Ontology-Layer Architecture

  • The ontology separates semantic knowledge, its object model, and runtime exposure; its typed content graph links Terms, Mappings, Constraints, and Evidence through semantic relations and structural references.The schema defines node fields, admissible relation types, and reference patterns, while allowing representational-capacity updates without changing instantiated content.Source: Agent-First Ontology-Layer Architecture · pp. 2–3
  • Two MCP tools retrieve semantic matches and resolve records with linked objects; a session manifest supplies compact source and usage information, while detailed records are fetched on demand.Only the manifest is placed in the prompt.Source: Agent-First Ontology-Layer Architecture · p. 3

Evidence-Grounded Ontology Initialization

  • The builder proposes workload-relevant concepts, probes raw sources for fields and linking paths, and commits only candidates supported by verification, retaining supporting records as Evidence.It uses the training workload and raw sources without observing gold answers; probes inspect types, values, and semantic consistency, and verification checks declared type, filter, and value-distribution requirements.Source: Evidence-Grounded Ontology… · p. 3

Trajectory-Grounded Ontology Evolution

  • Historical trajectories reveal effective or problematic ontology components; the evolution agent attributes recurrent signatures to content, tool, or schema and proposes localized single-level patches.Dependent content objects may change together when they implement the same hypothesis.Source: Trajectory-Grounded Ontology Evolution · pp. 3–4
  • A candidate update is retained only if it reaches the improvement margin against its parent on the same validation set, with identical decoding and interaction budgets for that backbone.Single-level changes isolate the attributed hypothesis; rejected candidates are not deployed, their outcomes are logged, and backbones evolve independently from the same initial state.Source: Trajectory-Grounded Ontology Evolution · p. 4

Experiments

EvoOntology improves benchmark performance across heterogeneous data tasks, and self-evolution adds gains beyond the builder-constructed ontology. The experiments compare against shared-scaffold baselines using reciprocal held-out evaluation.

Benchmarks

  • Experiments follow official protocols on DDR-Bench 10-K (Message-Wise, Trajectory-Wise), InsightBench (Insight, Summary), and BIRD Oracle Knowledge (EX, VES).DDR-Bench evaluates open-ended research across heterogeneous sources, InsightBench pairs business-intelligence flags with CSV data, and BIRD tests text-to-SQL questions over real-world databases.Source: Benchmarks · p. 4

Experimental Setup

  • All conditions share a ReAct scaffold, backbone, raw-data tools, decoding, and interaction budget; reciprocal A/B adaptation freezes the ontology before held-out testing.Baseline has no ontology, while Baseline + SL adds the builder-agent’s semantic layer as a static prompt fragment; held-out answers and evaluator feedback are excluded from construction and selection.Source: Experimental Setup · p. 4

Main Results

  • On DDR-Bench, Table 1 reports main results on the 10-K scenario; Table 2 compares against a memory-based persistence baseline averaged across four backbones, and ReAct + Memory injects the top-k past-trajectory episodes.Source: Table 1 · p. 5, Table 2 · p. 5
  • On InsightBench, EvoOntology improves both Insight and Summary on all four backbones, with a mean Insight gain of 1.9 points and a largest gain of +6.1 on DeepSeek-V4-Flash.Insight gains are smaller because short reference-style findings can saturate; Baseline + SL drops by −3.3 on Claude-Sonnet-5 Summary.Source: Table 3 · p. 5, Main Results · p. 5

Effect of Ontology Layer

  • Across DDR-Bench, InsightBench, and BIRD, evolution adds +7.7 Trajectory-Wise, +0.2 Insight, and +3.7 EX points beyond the Initial ontology.Initial ontology gains over Baseline are +12.3 points on DDR-Bench Trajectory-Wise, +0.8 on InsightBench Insight, and +5.1 on BIRD EX.Source: Effect of Ontology Layer · pp. 5–6

Main Results

  • On BIRD under Oracle Knowledge, EvoOntology improves EX by 7.4 points and VES by 8.6 points on average across every backbone.Baseline + SL raises VES across backbones but lowers EX by up to −5.6 on GPT-5.5; EvoOntology improves both metrics over both baselines and prior text-to-SQL systems.Source: Main Results · p. 5, Table 4 · p. 6

Analyses

Across DDR-Bench analyses of four backbones, EvoOntology improves through accepted rounds; ablations identify important loop steps, editable levels, and ontology object families, while cross-backbone transfer results show that backbone-matched stores perform best.

Effect of Iterative Evolution

  • All four backbones improve monotonically across accepted DDR-Bench rounds; GPT-5.6-sol reaches 93.5 Traj-Wise after five rounds and Claude-Opus-4.8 reaches 92.3 after four.Trajectories flatten in the last two rounds, consistent with recurrent cross-filing concepts becoming covered; the authors interpret the gains as converging refinement rather than a single fortunate patch.Source: Figure 4 · p. 6, Effect of Iterative Evolution · p. 6

Ablation Study on Evolution Loop

  • −11.2 Traj-Wise from removing Gate is the largest ablation drop, followed by Attribution (−6.3), Diagnose (−4.8), and typed Patch (−1.7).Without Gate, regressions can pass and may not be undone; without Attribution, edits can target the wrong level. The ablation disables each step in turn and averages final Evolved scores across four backbones.Source: Ablation Study on Evolution Loop · p. 6, Table 5 · p. 7
  • Tool-only evolution gains +13.2 over Baseline, above Content-only (+8.7) and Schema-only (+3.6), but below the full three-level loop (+20.0).The authors conclude that the three levels are complementary rather than substitutable; Tool-only provides the largest single-level gain, consistent with manifest reshaping being a dominant lever in the attribution analysis.Source: Ablation Study on Evolution Loop · pp. 6–7, Table 6 · p. 7

Ablation Study on Ontology Structure

  • −13.4 Traj-Wise from masking Mappings is the largest object-family drop, followed by Evidence (−8.7), Constraints (−3.5), and Relations (−2.1); Terms cannot be masked alone.Mappings ground Terms in concrete columns and join paths, while Evidence lets agents verify candidate SQL fragments against value distributions; every committed entry is required to be anchored in a probe query.Source: Table 7 · p. 7, Ablation Study on Ontology Structure · p. 7

Divergence across Backbones

  • No backbone pair exceeds 0.62 Term-identifier overlap; backbone-matched stores also achieve each column’s highest Traj-Wise score, while every off-diagonal drops by at least 6.6 points.Average diagonal-to-off-diagonal drops range from −6.6 for Sonnet-5 to −10.9 for GPT-5.5; the authors conclude that backbones develop different stores and benefit from backbone-specific evolution.Source: Figure 5 · p. 7, Divergence across Backbones · p. 7

Conclusion

EvoOntology is an automatically constructed, self-evolving ontology layer that data agents query at runtime through an MCP server. On benchmarks across six LLM backbones, it outperforms ReAct and traditional semantic-layer baselines.

  • EvoOntology consistently outperforms ReAct and traditional semantic-layer baselines on benchmarks across six LLM backbones.Source: Conclusion · p. 7
  • EvoOntology packages its automatically constructed ontology as an MCP server that data agents actively query at runtime.Source: Conclusion · p. 7
  • It refines the ontology through attribution-guided typed edits admitted only after a backbone-conditional paired evaluation gate.Source: Conclusion · p. 7

Content-Layer Growth across Evolution · Rounds

On DDR-Bench with GPT-5.6-sol, ontology content growth is concentrated in early accepted evolution rounds and stabilizes as recurrent semantic gaps are covered. Terms rise from 61 initially to 80 after five rounds, while growth across tracked elements falls below 5% per round after round three.

Rounds

  • 61 Terms in the Initial ontology increase to 80 after five accepted rounds, while every tracked element grows by less than 5% per round after round three.The analysis tracks Terms, Mappings, Constraints, Evidence, and instantiated Semantic Relations on DDR-Bench using GPT-5.6-sol.Source: Rounds · p. 10
  • Most content growth occurs in the first three accepted rounds, after which content-size curves flatten alongside Trajectory-Wise performance.The authors interpret early expansion as addressing recurrent semantic gaps, with content stabilizing once those gaps have been covered.Source: Rounds · p. 10

Cost of the Ontology Layer

The ontology layer adds modest per-turn context but reduces total task tokens by shortening trajectories; evolution further lowers cost while improving Trajectory-Wise performance.

Cost of the Ontology Layer

  • A compact manifest and MCP retrieval of detailed semantic records add per-turn context while reducing repeated schema discovery across the agent’s trajectory.Evolution strengthens this effect by improving how the agent discovers and grounds relevant semantics.Source: Cost of the Ontology Layer · p. 10
  • From Baseline to Initial ontology, input tokens per turn rise from 3.2K to 4.1K, turns per task fall from 14.6 to 11.2, and total tokens per task fall from 52.6K to 50.4K; Evolved reaches 8.4 turns, 42.0K tokens (~20% below Baseline), and Trajectory-Wise 89.5 versus 69.5.Table 8 reports DDR-Bench averages across the four-backbone analysis subset.Source: Table 8 · p. 10, Cost of the Ontology Layer · p. 10

Attribution across Editable Levels

Evolution gains are concentrated in Tool-level edits, with Content- and Schema-level edits contributing smaller shares; the three levels serve distinct, complementary roles.

Attribution across Editable Levels

  • Tool-level edits account for 57% of cumulative gain across six accepted rounds, Content-level edits 34% across eleven rounds, and Schema-level edits 9% across three rounds.The shares aggregate paired-evaluation improvements across four backbones. Tool edits improve how existing ontology content is exposed through the manifest and MCP tools; Content edits add or refine ontology elements identified from interaction trajectories; Schema edits change the ontology’s representational structure to address limits that instantiated-content changes cannot resolve.Source: Attribution across Editable Levels · p. 10

Case Study: Evolution of Card-Legality · Semantics

The case study shows how a Content-layer update adds reusable meaning for card legality status and its format-dependent condition, without changing the existing Tool or Schema layers. After paired validation, the agent can retrieve that interpretation through the ontology while native SQL applies and verifies the filter.

Semantics

  • The ontology could locate card and legality data, but left status values uninterpreted and legality’s dependence on game format implicit.Consequently, the agent had to rediscover these semantics from raw values during execution.Source: Semantics · pp. 10–11, Figure 8 · p. 11
  • The evolution agent attributes the gap to the Content Layer: existing browse and resolve tools and the Schema Layer could represent the needed knowledge.What was missing was a reusable description of the status field and its applicability condition.Source: Semantics · p. 11
  • The candidate adds a Legality Status Code Term grounded to legalities.status, Evidence about observed status values, and a Constraint requiring both status = ’Banned’ and format = target_format.The existing Card and Legality objects remain unchanged, with no Tool- or Schema-level modification.Source: Semantics · p. 11, Figure 8 · p. 11
  • After passing paired validation, the candidate becomes part of the Evolved ontology, adding the status interpretation and format condition as a localized Content-level update.Figure 8 depicts the accepted patch and the added or refined ontology objects.Source: Semantics · p. 11, Figure 8 · p. 11
  • Browse can surface Legality Status Code for banned- or legal-card queries, and resolve can return its physical Mapping, supporting Evidence, and format-dependent Constraint.These ontology tools expose the semantics the agent needs to formulate the task filter.Source: Semantics · p. 11
  • Native SQL execution still applies the filter and verifies returned records, while the ontology retains its structure and interface with the missing interpretation added.The case illustrates a specific semantic correction through a small connected set of objects.Source: Semantics · pp. 10–11

Related papers

Loading 2609.15779v1…