Source-linked AI summary
Wikontic: Constructing Wikidata-Aligned, Ontology-Aware Knowledge Graphs with Large Language Models
Alla Chepurova, Aydar Bulatov, Mikhail Burtsev, Yuri Kuratov
TL;DR
Current systems often treat KGs as retrieval scaffolds, leaving their quality as standalone knowledge resources underexplored. Wikontic builds ontology-aware KGs from text by extracting qualified triplets, applying Wikidata constraints, and normalizing entities. Its text-free QA is competitive across two datasets, and it achieves strong MINE-1 information-retention scores.
Problem
Many current open information extraction systems use KGs mainly to structure text retrieval, leaving their quality as compact, non-redundant knowledge resources underexplored.
Source: §1 · p. 2Method
Wikontic extracts triplets with qualifiers, applies Wikidata ontology constraints, and normalizes entity names to construct KGs from text.
Source: §1 · p. 2Results
Across HotpotQA and MuSiQue, Wikontic’s text-free QA is competitive with or surpasses several text-reliant baselines, while MINE-1 information-retention scores reach 84–86%.
Source: §4 · p. 9Takeaways & Limitations
Wikontic’s results support ontology-guided KG construction as a viable alternative to passage-level retrieval for multi-hop question answering.
Source: §4 · p. 9Experiments use proprietary OpenAI models and Llama 3.3-70B, and token efficiency does not measure end-to-end latency or throughput.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
Knowledge graphs (KGs) provide structured, verifiable grounding for large language models (LLMs), but current LLM-based systems commonly use KGs as auxiliary structures for text retrieval, leaving their intrinsic quality underexplored. In this work, we propose Wikontic, a multi-stage pipeline that constructs KGs from open-domain text by extracting candidate triplets with qualifiers, enforcing Wikidata-based type and relation constraints, and normalizing entities to reduce duplication. The resulting KGs are compact, ontology-consistent, and well-connected; on MuSiQue, the correct answer entity appears in 96% of generated triplets. On HotpotQA, our triplets-only setup achieves 76.0 F1, and on MuSiQue 59.8 F1, matching or surpassing several retrieval-augmented generation baselines that still require textual context. In addition, Wikontic attains state-of-the-art information-retention performance on the MINE-1 benchmark (86%), outperforming prior KG construction methods. Wikontic is also efficient at build time: KG construction uses less than 1,000 output tokens, about 3$\times$ fewer than AriGraph and $<$1/20 of GraphRAG. The proposed pipeline enhances the quality of the generated KG and offers a scalable solution for leveraging structured knowledge in LLMs.
1 Introduction
Wikontic addresses the underexplored quality of KGs as knowledge resources, rather than merely as scaffolds for text retrieval. It combines LLM extraction with Wikidata ontology constraints and entity normalization, producing KGs that support text-free multi-hop QA and retain information.
1 Introduction
- KGs represent knowledge as explicit subject-relation-object triplets, supporting verifiable queries, incremental updates, and multi-step reasoning.LLM representations are latent and often unverifiable, making them susceptible to hallucinations.Source: §1 · p. 1
- Closed information extraction relies on predefined entities and relations, while open extraction allows new names but can require costly retraining or yield inconsistent representations.Classical staged extraction can accumulate errors, and end-to-end systems may depend on scarce annotated corpora for adaptation.Source: §1 · pp. 1–2
- Most open information extraction pipelines use KGs mainly to scaffold text retrieval, leaving their quality as compact, non-redundant knowledge resources underused.Variation in entity and predicate surface forms can fragment graphs into redundant or inconsistent representations.Source: §1 · p. 2
- Wikontic integrates Wikidata ontology guidance into extraction, then refines triplets and normalizes entity names to build ontology-consistent KGs.Its six components include candidate extraction with qualifiers, ontology-aware refinement, entity normalization, KG storage, and retrieval for multi-hop QA.Source: §1 · p. 2
- 76.0 F1 on HotpotQA and 59.8 F1 on MuSiQue are achieved using the KG as the sole knowledge source, matching or surpassing several baselines that still rely on text.Wikontic also reaches 86% information-retention score on MINE-1.Source: §1 · p. 2
2 Methods: Wikontic
Wikontic constructs KGs by extracting qualified triplets, checking them against Wikidata ontology constraints, and normalizing entity names. It then uses the resulting KG for iterative multi-hop QA and evaluates graph quality through information retention, structural properties, and text-free QA.
2.1 Ontology and KG Databases
- Ontology and KG databases store Wikidata property constraints, entity types, taxonomies, labels, and aliases to support type compatibility, semantic alignment, linking, and deduplication.The schema includes 2,464 factual properties; type hierarchies propagate allowed properties from parent classes to their children.Source: §2.1 · pp. 3–4
2.2 Ontology-aware Triplet Extraction
- Triplet extraction captures qualifiers such as time, location, or conditions, then ontology refinement assigns types and selects relations compatible with Wikidata constraints.A final normalization stage links synonymous mentions to canonical KG labels and stores surface forms as aliases; triplets failing ontology checks are flagged but retained.Source: §2.2 · pp. 4–5
2.3 Retrieval for QA
- For multi-hop QA, the LLM iteratively retrieves KG neighborhoods, answers subquestions, and formulates the next subquestion for up to five steps.Each step links relevant entities to KG nodes and grounds its answer in the retrieved subgraph.Source: §2.3 · p. 5
2.4 Evaluation
- Evaluation tests text-free multi-hop QA using the same questions and candidate passages as HippoRAG and AriGraph, across MuSiQue and HotpotQA.The authors also assess compactness and deduplication; MINE-1 measures information retention but lacks complete ground-truth triplets, so it cannot measure classical precision or recall.Source: §2.4 · p. 5
- 84% average MINE-1 score across 100 articles, compared with 66% for KGGen and 47.80% for GraphRAG.The figure reports score distributions, with dotted vertical lines marking averages.Source: Figure 2 · p. 6
3 Results
Wikontic preserves factual information, builds connected and ontology-consistent graphs, and supports multi-hop QA using triplets alone. Its results show strong answer coverage and construction efficiency, while ablations identify contributions from multiple pipeline components.
3.1 Evaluations on MINE-1
- 86% with gpt-4.1-mini and 84% with gpt-4o exceed KGGen’s best score of 73% with Claude Sonnet 3.5 on MINE-1 retention.Wikontic has the highest retention performance across evaluated LLMs.Source: §3.1 · p. 6, Table 1 · p. 6
3.2 Graph Quality Analysis
- Full Wikontic yields the largest question neighborhoods; normalization yields highest density, refinement raises entities per relation, and relaxed ontology preserves greatest relation diversity.The graph-quality analysis also measures graph size, density, and diversity using structural statistics.Source: Table 2 · p. 6, Figure 3 · p. 7, §3.2 · p. 7
3.3 Answer Coverage
- Over 96% answer coverage for Wikontic and HippoRAG exceeds AriGraph’s 79.9%; Wikontic reaches 97.5% without ontology and 93.8% with 100% ontology entailment.The comparison uses the same gpt4o-mini configuration and 80 common MuSiQue test samples.Source: Table 3 · p. 7, §3.3 · pp. 7–8
3.5 Performance on QA Tasks
- 76.0 F1 on HotpotQA and 59.8 F1 on MuSiQue with triplets alone surpass several retrieval methods and are comparable to richer-context systems.Wikontic reasons over structured triplets without access to source documents.Source: Table 4 · p. 8, §3.5 · p. 9
3.4 Computational Efficiency
- 881 output tokens per paragraph is roughly three times fewer than AriGraph’s 2,500 and about twenty times fewer than GraphRAG’s 20,000.The evaluation compares token-based KG construction costs on MuSiQue.Source: Table 5 · p. 8, §3.4 · p. 8
3.6 Ablation Study
- Removing qualifiers costs 15.9 EM and 15.7 F1 on MuSiQue; ontology, normalization, aliases, and iterative subquestion reasoning contribute, with joint ontology/normalization removal worst.The single-step QA variant also performs significantly worse than the iterative version.Source: §3.6 · p. 9, Table 6 · p. 9
4 Conclusions
Wikontic constructs compact, internally consistent knowledge graphs from unstructured text by aligning triplets with the Wikidata ontology and deduplicating entities and relations. Its graphs support competitive text-free multi-hop question answering, retain source information, and use fewer construction tokens than comparison systems.
4 Conclusions
- Wikontic aligns extracted triplets with the Wikidata ontology and deduplicates entities and relations to produce compact, internally consistent graphs.With the KG as the sole knowledge source, it achieves competitive multi-hop QA performance against retrieval-augmented and KG-based baselines that use source texts.Source: §4 · p. 9
- For QA, Wikontic attains 64.5 EM / 76.0 F1 on HotpotQA and 46.8 EM / 59.8 F1 on MuSiQue, outperforming text-reliant systems and approaching larger text-dependent methods.Wikontic also achieves state-of-the-art MINE-1 information-retention scores of 84–86%, surpassing GraphRAG and KGGen.Source: §4 · p. 9
- On MuSiQue, Wikontic reaches 46.8 EM / 59.8 F1, outperforming text-reliant systems and approaching larger text-dependent methods; within a 10-hop subgraph, it maintains 70% answer coverage and denser local connectivity.Source: §4 · p. 9
- Under 1,000 output tokens, Wikontic uses about 3× fewer than AriGraph and 1/20 of GraphRAG while preserving accuracy.Only 3.5% of extracted triplets are flagged as ontology-misaligned; the pipeline can also operate without an ontology or integrate domain-specific ontologies, and its structured output is suitable for fine-tuning smaller task-specific models.Source: §4 · p. 9
- The authors conclude that ontology-aware KG construction can transform unstructured text into scalable, interpretable, and verifiable structured knowledge.They characterize this as bridging symbolic reasoning and generative language modeling.Source: §4 · p. 9
Limitations
The experiments are limited to four specified LLMs, and token efficiency measures model-generated construction tokens rather than latency or throughput. The current work focuses on Wikidata, though the pipeline can be adapted to other ontologies with matching constraint and entity-type formats.
Limitations
- Experiments use only GPT-4.1, GPT-4.1-mini, GPT-4omini, and Llama 3.3-70B.Token efficiency counts model-generated tokens during KG construction, reflecting provider billing but not end-to-end latency or throughput.Source: Limitations · p. 10
- Every pipeline stage currently uses LLMs with instructions and in-context examples, though future work could fine-tune smaller task-specific models on generated annotations.The authors state this could improve efficiency and lower computational cost.Source: Limitations · p. 10
- The current work focuses on Wikidata, while adaptation to another ontology requires a matching format for triplet constraints and entity types.Source: Limitations · p. 10
A.1 Reproducibility
The authors release materials for the pipeline and report experimental prompts and variability information. The released resources include stage prompts and code, ontology-building scripts, and a KG-only QA component.
A.1 Reproducibility
- Prompts and source code for each pipeline stage, Wikidata-derived ontology-building scripts, and the KG-only multi-hop QA component are released.Source: §A.1 · p. 12
- Metrics are averaged across multiple runs with standard deviations for Tables 4 and 6, while Table 3 standard deviations use bootstrap resampling.Source: §A.1 · p. 12
- Exact prompts used at each stage are provided in Appendix A.4 and A.5.Source: §A.1 · p. 12
A.2 Dataset Statistics
The evaluation uses 1,000-sample test splits for HotpotQA and MuSiQue, with smaller shared subsets for some comparisons. KG statistics are compared on 80 commonly available MuSiQue samples.
A.2 Dataset Statistics
- HotpotQA and MuSiQue test splits each contain 1,000 samples; KG statistics use 80 commonly available MuSiQue samples.QA results cover the whole evaluation set for Wikontic and HippoRAG, while AriGraph results cover 200 test samples.Source: §A.2 · p. 12
A.3 Computational resources
The reported total cost for KG construction and QA experiments was approximately $500. QA experiments were each repeated three times to calculate means and standard deviations.
A.3 Computational resources
- Approximately $500 covered KG construction and QA across datasets, models, and ablation studies.Each QA experiment was repeated three times to compute its mean and standard deviation.Source: §A.3 · p. 12
A.4 Prompts for triplet extraction
The prompts extract factual triplets from text, then refine their types and relations against Wikidata-style candidates before linking entity names to the existing graph. Qualifiers add context to triplets and remain attached rather than standing alone.
A.4 Prompts for triplet extraction
- Candidate triplets represent factual relationships as subjects, relations, and objects, with qualifiers attached to a triplet rather than treated as standalone triplets.The output format also includes subject and object types.Source: §A.4 · p. 12
- Extracted subject and object types are mapped to semantically similar Wikidata-style types, and prompts select the candidates that best fit the triplet and source text.The prompt asks for one selected type for each entity.Source: §A.4 · p. 13
- Relation candidates are restricted by which Wikidata-style relations can connect the chosen entity types, then selected for fit with the triplet and source text.Candidate relations are also mapped using semantic similarity.Source: §A.4 · p. 13
- For entity-name refinement, the prompt selects a contextually appropriate existing name or returns “None” when no suitable match exists.The matching name is returned exactly as it appears in the candidate list.Source: §A.4 · p. 14
A.5 Prompts for question answering
The question-answering prompts extract entities from questions, link them to entities in the constructed graph, and support stepwise decomposition of multi-hop questions. A final prompt checks whether the accumulated answers resolve the original question.
A.5 Prompts for question answering
- Question-answering prompts extract entities from a question and select relevant entities from the pre-constructed knowledge graph.The prompt sequence assigns these tasks to entity extraction and entity linking.Source: §A.5 · p. 14
- Question decomposition reformulates a multi-hop question using an already answered one-hop subquestion, advancing one reasoning hop at a time.The reformulated question has one fewer hop.Source: §A.5 · p. 15
- A separate prompt checks whether the subquestion-answer sequence resolves the original question, returning either its final answer or “NOT FINAL.”It returns “NOT FINAL” when further reasoning or hops are needed.Source: §A.5 · p. 15
A.6 Triplet extraction pipeline examples
The examples show a three-stage pipeline: extract triplets, refine them using ontology constraints, then link and normalize entities against the existing graph.
A.6 Triplet extraction pipeline examples
- Triplet extraction is followed by ontology-based validation and entity linking and normalization.Figure 4 presents these as the pipeline’s three stages.Source: Figure 4 · p. 16
- Ontology-based refinement retrieves candidate entity types from Wikidata’s type hierarchy, identifies relations allowed between those types, and reranks relations by semantic similarity.An LLM selects the final triplet configuration.Source: Figure 5 · p. 16
- Entity refinement compares each triplet’s subject and object with existing graph entities by type and semantic similarity, preserving a new entry when no match is selected.The process is described as reducing redundancy and supporting incremental graph updates.Source: Figure 6 · p. 16