Source-linked AI summary
Ontology-grounded Automatic Knowledge Graph Construction by LLM under Wikidata schema
Xiaohan Feng, Xixin Wu, Helen Meng
TL;DR
Manual KG construction is costly to scale, and direct LLM outputs can lack schema consistency and integration with existing knowledge bases. The paper uses competency questions and Wikidata-aligned ontology grounding to construct graphs; on benchmark datasets it reports strong constrained-schema comparisons alongside a small unconstrained-setting regression.
Problem
Manual KG curation is time-consuming, expensive, and hard to scale, while direct LLM construction faces schema-consistency, coverage, and knowledge-base alignment challenges.
Source: §1 · p. 1Method
The pipeline generates competency questions, aligns extracted relations with Wikidata properties, and grounds LLM construction of RDF triples in the resulting ontology.
Source: §1 · p. 2, §3.4 · p. 4Results
Under target-schema constraints, the method exceeds all baselines on Wiki-NRE and SciERC and remains competitive with fine-tuned SOTA on WebNLG, but shows a small regression without schema constraints.
Source: §4.2 · p. 6Takeaways & Limitations
The authors suggest Wikidata-conformant KGs could support an interpretable QA system with access to both common and proprietary knowledge.
Source: §5 · p. 7The pipeline uses three LLM calls per document plus one per extracted relation, and its resource-cost comparison with tuned non-LLM baselines is unclear.
Source: §4.3.3 · p. 7
Abstract
from arXiv · showhide
We propose an ontology-grounded approach to Knowledge Graph (KG) construction using Large Language Models (LLMs) on a knowledge base. An ontology is authored by generating Competency Questions (CQ) on knowledge base to discover knowledge scope, extracting relations from CQs, and attempt to replace equivalent relations by their counterpart in Wikidata. To ensure consistency and interpretability in the resulting KG, we ground generation of KG with the authored ontology based on extracted relations. Evaluation on benchmark datasets demonstrates competitive performance in knowledge graph construction task. Our work presents a promising direction for scalable KG construction pipeline with minimal human intervention, that yields high quality and human-interpretable KGs, which are interoperable with Wikidata semantics for potential knowledge base expansion.
1. Introduction
Manual KG curation is costly and difficult to scale, while direct LLM construction risks inconsistency, incomplete coverage, and misalignment with standard ontologies. The paper proposes grounding LLM-generated graphs in an ontology aligned with Wikidata and reports benchmark improvements in its contribution summary.
1. Introduction
- Expert-led KG construction is time-consuming, expensive, and difficult to scale across large, evolving domains.The introduction identifies automatic extraction of structured knowledge from unstructured sources as a need.Source: §1 · p. 1
- Direct LLM construction can produce inconsistent or redundant facts, incomplete or training-data-biased graphs, and outputs misaligned with standard ontologies.The introduction links these challenges to the lack of a unified schema and possible gaps in pretraining coverage, including proprietary documents.Source: §1 · pp. 1–2
- The proposed pipeline generates competency questions and answers, derives an ontology by matching relations to Wikidata properties, then grounds KG construction in that ontology.The schema can be extended when needed, and the approach aims to support interpretability and interoperability with public knowledge bases.Source: §1 · p. 2
- The authors report benchmark experiments showing improved KG quality over traditional methods, alongside interpretability and utility of generated graphs.The listed contributions describe a pipeline combining competency-question generation, ontology alignment, and KG grounding.Source: §1 · p. 2
2. Literature Review
Prior KG construction work spans rule-based, neural, distant-supervision, semi-supervised, and LLM-based methods, with reported trade-offs in noise, consistency, portability, and interpretability. The paper positions Wikidata-grounded ontology construction as a response to these remaining issues.
2. Literature Review
- Earlier KG construction approaches used rules and hand-crafted features, followed by neural methods described as more flexible and scalable.Source: §2 · p. 2
- Distant supervision scales relation-extraction training data by using entity co-occurrence and knowledge-base relations, but can suffer from noise and incomplete coverage.Source: §2 · p. 3
- Unsupervised and semi-supervised methods reduce reliance on labeled data but often struggle with consistency and quality control.Source: §2 · p. 3
- LLM-based methods can generate triples, but the review says missing canonicalization hinders portability and interoperability, while vector-similarity approaches fall short in interpretability.The paper presents Wikidata-schema grounding as addressing these issues and reports decent performance on private documents.Source: §2 · p. 3
3. Method: Ontology-grounded KG Construction
The method builds a KG through four stages: competency-question generation, relation extraction and ontology matching, ontology formatting, and KG construction. It uses LLMs throughout, with Wikidata properties grounding the ontology and the final triples.
3. Method: Ontology-grounded KG Construction
- The pipeline comprises competency-question generation, relation extraction and ontology matching, ontology formatting, and KG construction.Figure 1 is described as an overview of this four-stage approach.Source: §3 · p. 3
3.1. Competency Question (CQ)-Answer Generation
- LLMs generate competency questions and answers from input documents to scope the KG around domain information needs and intended use cases.User-submitted domain-defining questions can also expand the ontology scope incrementally.Source: §3.1 · p. 3
3.2. Relation Extraction and Ontology Matching
- Extracted CQ relations are matched to Wikidata properties using description embeddings and top-1 retrieval, then LLM-vetted; unmatched properties may be retained or discarded depending on whether schema expansion is allowed.The authors report that even 7B/14B models spontaneously recalled Wikidata knowledge in preliminary prompting experiments.Source: §3.2 · p. 4
3.3. Ontology Formatting
- The ontology is formatted in OWL using Wikidata descriptions, domains, and ranges, while new properties receive LLM-inferred domain and range classes.The authors state this grounds the KG in a machine-readable ontology close to Wikidata semantics.Source: §3.3 · p. 4
3.4. KG Construction
- For each competency-question and answer pair, the LLM extracts relevant entities, maps them to ontology properties, and outputs RDF triples as the KG.Source: §3.4 · p. 4
4. Experiments and Discussion
Across Wiki-NRE, SciERC, and WebNLG, the pipeline is evaluated with target-schema-constrained and unconstrained settings. It exceeds all baselines on Wiki-NRE and SciERC when constrained, while richer unconstrained ontologies create a trade-off with benchmark scores and remain partly unmeasured.
4.1. Experiment settings
- Experiments use Wiki-NRE, SciERC, and WebNLG, comparing target-schema-constrained evaluation with an unconstrained setting that retains matched ontology properties outside the dataset schema.The datasets contain relation types not equivalent to Wikidata properties, particularly in SciERC; Wiki-NRE results use a 1,000-sample test subset, while SciERC and WebNLG use 974 and 1,165 samples, respectively.Source: §4.1 · p. 5
4.2. Result
- Under target-schema constraints, the pipeline exceeds all baselines on Wiki-NRE and SciERC and remains competitive with fine-tuned SOTA on WebNLG.The authors also report performance improvement when using GPT-4o.Source: Table 1 · p. 6, §4.2 · p. 6
4.3.1. Performance discrepancy on different grounding ontology
- Without schema constraints, richer discovered ontologies lower scores against limited target schemas, while additional properties cannot be directly evaluated because test annotations do not cover them.The authors identify large document sets with limited knowledge scope and concise schemas as the best fit, and leave ontology trimming, fine-tuning, or pretraining as possible future directions.Source: §4.3.1 · p. 6
4.3.2. Utility of generated KG
- The authors propose that explicit KGs could support an interpretable QA system in which users can understand and verify the KG and retrieval query.They also describe Wikidata schema use as offering potential interoperability with the Wikidata knowledge base.Source: §4.3.2 · p. 7
4.3.3. Computational resources
- The pipeline uses three LLM calls per document plus one per extracted relation, and its carbon footprint is not straightforward to compare with tuned non-LLM baselines.The smallest model, Mistral-7B, has over 10x the parameter size of the T5 models used by those baselines; the authors note a possible zero-shot resource-cost advantage for small document collections without training. Cited LLM baselines use one and two calls per document, respectively, but target triplets rather than a formatted ontology.Source: §4.3.3 · p. 7
5. Conclusion
The conclusion presents the ontology-grounded pipeline as a way to construct KGs across domains with competitive performance. Wikidata-conformant generated KGs could also support an interpretable QA system with access to common and proprietary knowledge.
5. Conclusion
- The authors report that their ontology-grounded LLM approach constructs high-quality KGs across domains while maintaining competitive performance with state-of-the-art baselines.The approach leverages Wikidata's structured knowledge and grounds KG construction in a generated ontology.Source: §5 · p. 7
- The proposed pipeline grounds KG construction in a generated ontology and leverages Wikidata's structured knowledge.Source: §5 · p. 7
- Generated KGs conformant with the Wikidata schema could support an interpretable QA system with access to common and proprietary knowledge.Source: §5 · p. 7
A. Sample generated KG
The no-schema-constraint example generates a KG for Mohammad Firouzi, but evaluation is penalized because official annotations include only place of birth and nationality.
A. Sample generated KG
- The example KG includes occupation, citizenship, birthplace, and birth date, while official annotations contain only birthplace and nationality, leading to low precision.The sample is explicitly generated without a schema constraint.Source: A. Sample generated KG · p. 11
B. Preprocessing of Wikidata schema
Wikidata properties are restricted to common data types to conserve LLM context and mitigate performance drops with large ontologies; property identifiers are replaced by literal labels.
B. Preprocessing of Wikidata schema
- To save input context and mitigate performance drops with large ontologies, the pipeline keeps item, quantity, string, monolingual-text, and point-in-time properties.Property identifiers such as P19 are replaced with PascalCase labels such as PlaceOfBirth to align with common LLM pretraining objectives.Source: B. Preprocessing of Wikidata schema · p. 11
C. Prompts
The prompts reuse a common format across datasets and guide a sequence from competency questions and answers to relation extraction, ontology matching, ontology formatting, and KG generation.
C. Prompts
- Prompts are reused across datasets.Source: C. Prompts · p. 12
C.1. CQ generation
- CQ generation requests up to three document-grounded questions based on abstract-level concepts, with the limit adjustable.The prompt says the cap is chosen for efficiency and may be adjusted.Source: §C.1 · p. 12
C.2. CQ answering
- The CQ-answering prompt directs the model to answer from the document and admit when it does not know.Source: §C.2 · p. 13
C.3. Relation extraction
- Relation extraction asks for properties between entities and literals, excludes entities or entity classes, and merges extracted relations and concepts into lists.The prompt cautions that not every competency question contains a valid property.Source: §C.3 · p. 13
C.4. Ontology matching
- Ontology matching labels properties as similar when they are semantically similar or inverse, and requires a yes-or-no answer.Source: §C.4 · p. 15
C.5. Ontology formatting
- For Wikidata properties, ontology formatting retrieves descriptions, domains, and ranges; other properties are formatted into RDF with inferred classes and usage comments.The prompt specifies Wikidata property namespaces and Turtle syntax.Source: §C.5 · p. 15
C.6. KG generation
- The KG-generation prompt maps relations from question-answer pairs and documents to the provided ontology, forbids new properties, and requests Turtle output.The Douglas Adams example’s resulting KG includes relations such as occupation, education, notable works, memberships, and awards.Source: §C.6 · pp. 16, 18, 19