PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoSchema: Live Schema Grounding for Agentic Text-to-Sparql over Heterogeneous Knowledge Graphs

Yiming Zhang, Koji Tsuda

arXiv:2608.14228v1cs.LG

TL;DR

Heterogeneous knowledge-graph schemas make reliable text-to-SPARQL difficult, while curated schema files require costly maintenance. AutoSchema grounds query decisions in live endpoint evidence and consistently improves mean factoid accuracy over TogoMCP across biomedical evaluations.

  • Problem

    Heterogeneous resources use differing schemas, identifiers, and links, making reliable natural-language-to-SPARQL querying technically demanding and curated schema maintenance costly.

  • Method

    AutoSchema provides training-free live schema grounding through source selection, entity resolution, relation-path discovery, and cross-resource bridging during agentic query construction.

  • Results

    AutoSchema consistently improves mean factoid accuracy over TogoMCP across biomedical KGQA evaluations and improves BioASQ performance while reducing average tool use.

  • Takeaways & Limitations

    Live endpoint evidence is a practical grounding interface for knowledge-graph agents, including irregular and previously undocumented graphs.

  • Takeaways & Limitations

    The findings do not establish live schema grounding as a complete solution to automatic federation; bridge selection was rare, multi-resource gains were noisy, and transfer used one run.

Abstract

from arXiv · show

Life science knowledge graphs make large collections of structured data available through SPARQL, but each resource uses its own schema, identifiers, and links. TogoMCP helps language model agents query these resources by providing curated Metadata Interoperability Exchange files. Creating and maintaining these files still requires language model assisted drafting, validation, and manual review. We study \emph{live schema grounding}, where an agent obtains the schema evidence needed for a question directly from the current endpoints. We present \textsc{autoschema}, a general framework for live schema grounding that requires no training. It inspects live schemas, maps entity names in a question to graph identifiers, explores relation paths, and finds possible connections between resources during iterative query construction. We use TogoMCP as our main comparison framework. We evaluate \textsc{autoschema} on Resource Focused Biomedical KGQA, Multi Resource Biomedical KGQA, Longitudinal Biomedical Semantic QA over BioASQ Task B, and Chemistry Knowledge Graph Transfer to a previously undocumented RDF graph. \textsc{autoschema} improves mean factoid accuracy over TogoMCP in the biomedical KGQA tasks and gives consistent gains in the longitudinal BioASQ evaluation. It also reduces iteration budget exhaustion and uses fewer tool calls on average in the core evaluation. The transfer study gives preliminary evidence that live schema grounding can support irregular and previously unseen graphs without first creating a curated schema file.

1 Introduction

The paper introduces live schema grounding: agents discover current endpoint evidence at question time to construct executable SPARQL over heterogeneous knowledge graphs. It presents AutoSchema as a training-free framework and reports improved biomedical accuracy, lower iteration exhaustion, fewer tool calls, and preliminary transfer to an irregular new graph.

  • Motivation: MIE creation combines automatic discovery, language-model-assisted drafting, endpoint validation, and manual review, creating privacy and maintenance constraints.Hosted model APIs may be unsuitable for private RDF data, while exploring and drafting schema context requires substantial upkeep.
  • Framework: AutoSchema inspects classes and properties, resolves question entities to graph identifiers, explores local paths, and bridges resources during iterative query construction.The agent requests evidence as needed, generates and executes SPARQL, then uses results to guide the next step.
  • Evaluation: Across open-weight models, AutoSchema improves mean factoid accuracy over TogoMCP in biomedical KGQA, with consistent BioASQ factoid gains, lower iteration exhaustion, and fewer average tool calls.TogoMCP provides MIE files containing schema descriptions, examples, errors, and cross-resource query patterns, making it the main baseline.
  • Evaluation: The chemistry transfer study preliminarily suggests live grounding supports a previously undocumented graph with an irregular schema.The evaluation also covers resource-focused and multi-resource biomedical KGQA plus longitudinal BioASQ semantic QA.

2 Related Work

Prior text-to-SPARQL and KGQA work addresses entity linking, relation selection, schema retrieval, iterative graph feedback, and federated querying, but typically assumes a known target graph or relies on precomputed summaries. AutoSchema instead uses live, question-specific endpoint observations as grounding evidence, with evaluation spanning heterogeneous biomedical resources and an undocumented chemistry graph.

  • Text-to-SPARQL and KGQA: Text-to-SPARQL research centers on entity linking, relation selection, and compositional reasoning, while benchmark protocols may overestimate transfer to unseen relations and compositions.Benchmarks include WebQuestionsSP, ComplexWebQuestions, LC-QuAD 2.0, CFQ, and KQA Pro.
  • Text-to-SPARQL and KGQA: Recent KGQA systems retrieve graph context, constrain program actions, or bind logical forms to graphs, generally assuming a known target graph.SG-KBQA shows that schema context can help with unseen graph elements.
  • Iterative and federated grounding: Tool-based systems make grounding iterative through structured-data interfaces, graph feedback, reasoning paths, schema evidence, endpoint discovery, and federated query formulation.The controlled comparison fixes the agent and execution tools while changing only the grounding-evidence source.
  • Iterative and federated grounding: RDF summaries and federated engines provide dataset descriptions, graph-structure inspection, source selection, or query execution, whereas AutoSchema converts live, question-specific observations into agent grounding evidence.AutoSchema does not replace a federated query engine.
  • Biomedical Linked Data: Biomedical Linked Data is challenging because resources use different schemas and identifiers, motivating systems ranging from Bio2RDF and BioFed to TogoMCP’s direct LLM access to endpoints.Related resources include Bio-SODA, the RDF Portal, and RDF-config’s expert schemas behind a fixed query builder.
  • Evaluation settings: The evaluation combines graph-grounded Resource Focused and Multi Resource Biomedical KGQA, multi-year BioASQ Task B questions, and transfer to the tmQM-RDF chemistry graph.BioASQ includes factoid, list, and yes-or-no questions but does not guarantee that every answer exists in a given graph.

3 Method

AutoSchema grounds agentic SPARQL construction from live endpoints rather than assumed schemas, using modular evidence for source selection, entity resolution, path discovery, and cross-database bridging. Its compact, query-specific grounding supports iterative construction while requiring bridge candidates to be executed and inspected before use.

  • Framework: AutoSchema observes live endpoints when a question arrives and returns only the schema evidence needed for that question.It addresses differences in names, identifiers, relations, and inter-graph links across independently maintained knowledge graphs.
  • Framework: The framework sits between the agent and live knowledge graphs, supplying compact endpoint views while leaving query generation and execution to the agent.It is independent of a particular agent or execution system.
  • Core modules: Four modules ground source selection, entity identifiers, relation paths, and cross-source connections during iterative query construction.The modules progressively move from broad source discovery to focused entity and path evidence.
  • Source summarization: Source summarization samples classes, predicates, object kinds, example IRIs, and values, then returns named-graph scope and an executable example query.The sampled view helps distinguish literal identifiers from IRI-valued links without requiring a complete ontology.
  • Entity resolution: Entity search returns stable URI subjects matched through labels, identifiers, alternative labels, or biomedical accession forms, while excluding blank nodes.The agent uses question context and the source summary to choose among candidate IRIs.
  • Cross-database bridging: Cross-database bridging checks identifier conversion, direct IRI links, and shared properties, but returns join candidates that the agent must execute and inspect.Direct IRI reuse is strongest; shared property names are weaker because their meanings may differ.

4 Experiments … 4.5 Biomedical Knowledge Graph QA

The experiments compare TogoMCP with autoschema across biomedical QA and chemistry transfer tasks under a shared evaluation protocol. Autoschema achieves the strongest reported biomedical QA results while reducing interaction-limit exhaustion, though Multi Resource accuracy gains are directional rather than conclusive.

  • 4.1 Research Questions: The evaluation asks whether live grounding improves biomedical KGQA, transfers to longitudinal and chemistry settings, and changes interaction cost.
  • 4.2 Compared Frameworks: TogoMCP and autoschema use the same models, questions, endpoints, and interaction budgets, while autoschema operates without an MIE file.Single shot is an autoschema ablation allowing at most one call to each grounding module per question.
  • 4.3 Evaluation Tasks: The tasks progress from guaranteed graph-answer questions to open biomedical questions and a graph outside the development domain.
  • 4.3 Evaluation Tasks: Resource Focused and Multi Resource Biomedical KGQA use graphlet-anchored questions, with the latter requiring verified links across resources.Questions begin from connected fragments sampled from real RDF triples, whose gold SPARQL queries are reconstructed and executed before natural-language generation.
  • 4.3 Evaluation Tasks: Longitudinal Biomedical Semantic QA uses factoid, list, and yes-or-no questions from six yearly BioASQ Task B test sets, while chemistry transfer uses the unseen tmQM-RDF graph.The longitudinal task also tests whether agents choose graph tools when appropriate; summary questions are excluded.
  • 4.4 Models, Metrics, and Protocol: The study evaluates gpt-oss-120b and gemma4-31b locally with vLLM on one NVIDIA A100 GPU, using a 20-iteration limit and reporting limit rate.Results are generally means ± standard deviations over three runs, with factoid top-1 accuracy, list F1, and yes-or-no accuracy as metrics.
  • 4.5 Biomedical Knowledge Graph QA: Autoschema gives both models the highest factoid accuracy and list F1 in Resource Focused Biomedical KGQA while lowering limit rate relative to TogoMCP.The gains are larger for gpt-oss-120b, which reaches the interaction limit much more often under the other frameworks.
  • 4.5 Biomedical Knowledge Graph QA: Autoschema again has the highest mean factoid accuracy for both models and a lower limit rate in Multi Resource Biomedical KGQA, but accuracy gains are directional because run variation is larger.Trace analysis attributes many solutions to shared reference values found through local entity expansion; the dedicated bridge module is rarely selected.

4.6 Longitudinal Biomedical Semantic QA … 4.11 Qualitative Analysis

Across longitudinal BioASQ tests, cost analyses, transfer experiments, ablations, and trace studies, autoschema generally improves factoid retrieval and reduces interaction burden, while its strongest benefits appear when graph structure is uncertain or irregular. The evidence also shows limits: latency and some answer types remain variable, and chemistry transfer results are preliminary.

  • 4.6 Longitudinal Biomedical Semantic QA: Autoschema improves mean factoid accuracy over TogoMCP on both evaluated BioASQ years, with clearer gains on Task 8B than Task 13B.List and yes-or-no results are mixed, indicating the main benefit is more reliable retrieval of short entity answers.
  • 4.7 Results Across BioASQ Task B Years: Mean factoid accuracy is higher with autoschema than TogoMCP in every one of the six yearly BioASQ Task B test sets.Gain sizes vary and run variation makes some individual years less conclusive, but the consistent direction supports temporal robustness.
  • 4.8 Cost and Latency: Autoschema uses fewer tool calls than TogoMCP for both models while latency remains comparable across the pooled biomedical and BioASQ evaluations.Trace inspection suggests early schema lookup can replace several failed SPARQL attempts based on guessed predicates.
  • 4.9 Chemistry Knowledge Graph Transfer: In the preliminary one-run chemistry transfer, autoschema improves factoid accuracy and lowers the limit rate but uses more time and tool calls, while List F1 is unchanged.Both frameworks reach the same factoid accuracy on a separate flat control graph, suggesting the benefit is concentrated in irregular schemas.
  • 4.10 Ablations: Most iterative-grounding benefits arise from the first grounding request on Resource Focused Biomedical KGQA, whereas follow-up grounding can matter when a second source emerges in Multi Resource Biomedical KGQA.Results remain similar for gemma4-31b, but gpt-oss-120b loses list performance under the limit in the multi-resource task.
  • 4.11 Qualitative Analysis: In paired traces, autoschema resolves the collection record and discovers the local path to the phylum, while TogoMCP follows an absent relation pattern and reaches the interaction limit.Chemistry traces likewise recover mismatched namespaces and nested property paths; on a flat control graph, both frameworks return the same answer in the same number of calls.

5 Conclusion

The conclusion presents AutoSchema’s live schema grounding as a practical alternative to prewritten schema files, with consistent biomedical gains and preliminary transfer evidence. It also identifies limits around federation and evaluation breadth, motivating stronger planning, verification, and broader testing.

  • Contribution: AutoSchema implements live schema grounding for agentic text-to-SPARQL without requiring a prewritten, manually reviewed schema file.Its four capabilities expose source structure, entity candidates, local relation paths, and potential cross-database connections as needed.
  • Results: Across two open-weight models, AutoSchema consistently improves mean factoid accuracy over TogoMCP on two biomedical KGQA tasks.It also improves gpt-oss-120b across six yearly BioASQ Task B test sets and reduces average tool use in biomedical evaluation.
  • Transfer: The tmQM-RDF transfer study provides preliminary evidence that AutoSchema can expose a previously undocumented graph without first producing a curated schema file.This transfer experiment contains only one run.
  • Limitations and future work: The findings support live endpoint evidence as a practical grounding interface, but not a complete solution to automatic federation.The dedicated bridge capability was rarely selected, some multi-resource gains were noisy, and future work should strengthen source and bridge planning, answer verification, and evaluation on unseen graphs.

Ethics and Privacy Statement

The study uses public questions and RDF endpoints without human participants or personal data, but endpoint lookups may expose question terms and impose service load. AutoSchema can produce incorrect or incomplete answers, so it is intended for research and information retrieval—not clinical decision making—and outputs should be checked against source evidence.

  • The study uses public benchmark questions and accessible RDF endpoints, involves no human participants, and collects no personal data.
  • Question-dependent lookup queries may reveal scientific terms to endpoint operators and add load to public services.Deployments handling sensitive questions should use access-controlled endpoints and avoid exposing private information.
  • AutoSchema may return incomplete or incorrect answers because endpoint content, entity matches, or generated queries can be wrong.This risk is especially important for biomedical questions; answers should be checked against returned source evidence before practical use.
  • AutoSchema is intended for research and information retrieval, not clinical decision making.Experiments use locally hosted open-weight models, avoiding transmission of questions to a closed model API.

A AutoSchema Tool Interfaces

AutoSchema is a standalone MCP server whose core interface provides four asynchronous, language-model-independent functions for querying configured RDF endpoints. Agents use the returned evidence to construct queries, while execution remains external to the framework.

  • Implementation: AutoSchema exposes four asynchronous core functions through a standalone MCP server.The functions query RDF endpoints and do not call a language model.
  • Implementation: Deployments map database arguments to endpoints and optionally named graphs, or accept an endpoint URL directly.Each database argument is associated with one endpoint and zero or more named graphs.
  • Implementation: Query execution remains outside AutoSchema, with agents sending constructed queries to an executor available in their environment.The agent builds queries from the evidence returned by AutoSchema.

A.1 Live Index and Output Scope

The live index uses bounded, cached endpoint surveys to provide observed schema evidence for query construction. Because coverage is sampled and failures are skipped, missing classes or properties do not establish that they are absent.

  • Index construction: The implementation surveys at most 25 classes and 40 properties per class, sampling up to 300 class instances for property discovery.Requests use bounded concurrency, and the resulting index is cached in memory and on disk; deployments can rebuild it when sources change.
  • Output scope: The interface returns observed evidence rather than a complete ontology, so entities outside the bounded sample may be omitted.An absent class or property in the summary may still exist at the endpoint.
  • Output scope: Individual endpoint failures are skipped, allowing index construction to continue but preventing the tools from proving that unobserved relations are absent.The tools support forming and testing queries rather than establishing relation nonexistence.

A.2 Source Catalog Adapter

The Source Catalog Adapter supports standalone querying against one endpoint or a registry of named endpoints. It provides catalog operations for selecting sources and derives metadata when catalog records are unavailable.

  • Source Catalog Adapter: The standalone server supports either a single endpoint or a registry of named endpoints.The evaluation uses many named biomedical sources.
  • Source Catalog Adapter: Three catalog calls—find_databases, list_databases, and list_categories—select a source before the four core tools are used.
  • Source Catalog Adapter: Existing source titles and descriptions are retained when available, while missing catalog records receive entries derived from live-index class and property names.

A.3 Evaluation Prompt Policy

The evaluation policy directs agents to ground graph questions through live MCP interfaces rather than precautionary RDF access or prewritten schema files. It constrains source inspection, entity and relation exploration, query execution, failure recovery, and answer completion under interaction limits.

  • A.3 Evaluation Prompt Policy: The wrapper converts MCP schemas into function calls, returns textual tool results, and limits each result to 6,000 characters in the model context.The AutoSchema system prompt then applies the evaluation policy.
  • A.3 Evaluation Prompt Policy: Agents first classify whether a question requires structured graph data, then call find_databases once and get_rdf_schema once per selected source.General facts and literature questions do not open RDF sources merely as a precaution.
  • A.3 Evaluation Prompt Policy: Agents use search_rdf_entity for name-to-IRI mapping, expand_entity for indirect or blank-node values, and call the bridge tool once before a genuine two-source join.These steps provide live grounding for entity resolution and cross-resource connections.
  • A.3 Evaluation Prompt Policy: Agents execute candidate SPARQL with the required named-graph scope, pivot after two failed or empty attempts, and return the best supported answer when the budget is nearly exhausted.The policy favors changing the predicate, source, or tool over repeating the same attempt, followed by a concise answer once sufficient evidence is found.
Loading 2608.14228v1…