Source-linked AI summary
ODKE+: Ontology-Guided Open-Domain Knowledge Extraction with LLMs
Samira Khorshidi, Azadeh Nikfarjam, Suprita Shankar, Yisi Sang, Yash Govind, Hyun Jang, Ali Kasgari, Alexis McClimans, Mohamed Soliman, Vishnu Konda, Ahmed Fakhry, Xiaoguang Qi
TL;DR
Keeping knowledge graphs comprehensive and current is costly, motivating automated extraction that can meet production needs. ODKE+ uses ontology-guided extraction and type constraints, and reports higher post-ranking precision alongside improved coverage relative to third-party sources.
Problem
Maintaining comprehensive, up-to-date knowledge graphs is costly and challenging.
Source: §1 · p. 1Method
ODKE+ uses tailored ontology snippets for schema-aware extraction and validates extracted rules against KG type constraints.
Source: §3.3 · pp. 4–5Results
98.8% post-ranking factual precision, up from 91% for raw LLM extraction, accompanies improved coverage relative to third-party knowledge sources.
Source: §4.1 · p. 6, §5 · p. 6Takeaways & Limitations
The deployed system supports batch and real-time use cases while extracting and ingesting facts across 195 predicates.
Source: §5 · p. 6Because ODKE+ relies on LLMs trained on broad web data, it may propagate social or cultural biases that grounding and corroboration do not resolve.
Source: Ethics Statement · p. 7
Abstract
from arXiv · showhide
Knowledge graphs (KGs) are foundational to many AI applications, but maintaining their freshness and completeness remains costly. We present ODKE+, a production-grade system that automatically extracts and ingests millions of open-domain facts from web sources with high precision. ODKE+ combines modular components into a scalable pipeline: (1) the Extraction Initiator detects missing or stale facts, (2) the Evidence Retriever collects supporting documents, (3) hybrid Knowledge Extractors apply both pattern-based rules and ontology-guided prompting for large language models (LLMs), (4) a lightweight Grounder validates extracted facts using a second LLM, and (5) the Corroborator ranks and normalizes candidate facts for ingestion. ODKE+ dynamically generates ontology snippets tailored to each entity type to align extractions with schema constraints, enabling scalable, type-consistent fact extraction across 195 predicates. The system supports batch and streaming modes, processing over 9 million Wikipedia pages and ingesting 19 million high-confidence facts with 98.8% precision. ODKE+ significantly improves coverage over traditional methods, achieving up to 48% overlap with third-party KGs and reducing update lag by 50 days on average. Our deployment demonstrates that LLM-based extraction, grounded in ontological structure and verification workflows, can deliver trustworthiness, production-scale knowledge ingestion with broad real-world applicability. A recording of the system demonstration is included with the submission and is also available at https://youtu.be/UcnE3_GsTWs.
1 Introduction
Knowledge graphs support downstream applications, but keeping them comprehensive and current remains difficult. The section motivates automation against demands of scale, varied source formats, trust, and update speed.
1 Introduction
- KG quality and completeness directly influence downstream applications including question answering, entity disambiguation, and relationship extraction.Ontologies define the entity types, properties, and relationships that govern how knowledge is stored and interpreted.Source: §1 · p. 1
- Human curation is costly and labor-intensive, making it difficult to maintain knowledge graphs that are both comprehensive and up-to-date.The section calls for automated updates that preserve precision, correctness, and consistency.Source: §1 · p. 1
- The system must scale to billions of web pages and millions of Wikipedia articles while handling information in text, infoboxes, and web tables.These sources require flexible extraction techniques.Source: §1 · p. 1
- Noisy, conflicting, outdated, and frequently changing web facts make evidence selection, trust assessment, and periodic or near-real-time updates necessary.High-interest topics require both periodic and near-real-time updates.Source: §1 · p. 2
2 Background
Prior knowledge-graph systems leave gaps in freshness, evidence verification, and ontology alignment, while traditional rule-based extractors can be difficult to scale and generalize. ODKE+ presents a unified production-oriented approach and extends earlier ODKE versions.
2 Background
- Rule-based extractors for semi-structured sources such as Wikipedia infoboxes are difficult to scale and generalize.The section frames this alongside the challenge of providing one production-grade solution to volume, variety, veracity, and velocity.Source: §2 · p. 2
- Prior LLM-based KG systems often lack streaming updates, corroboration of source evidence, or ontology-guided prompting, while search-based completion lacks broad coverage and modern LLM extraction.The cited prior-work discussion describes these as recurring limitations, not as universal properties of every system.Source: §2 · p. 2
- ODKE+ combines batch and streaming modes with ontology-guided prompting, evidence retrieval, hybrid extraction, corroboration, normalization, and KG ingestion.The demo follows the pipeline from identifying missing or outdated facts through near-real-time KG updates.Source: §2 · p. 2
2.1 Comparison with Prior Versions
- ODKE v1 began as Saga’s extraction component, and ODKE v2 added multilingual support, streaming ingestion, and limited LLM-based extractors.The paper presents ODKE v3 (ODKE+) as adding lightweight-LLM grounding verification, dynamic ontology-guided prompting, and expanded predicate support.Source: §2.1 · p. 2
3 System Architecture
ODKE+ uses a modular pipeline to select entities and evidence, extract candidate facts, and corroborate them before KG updates. Its extraction combines pattern rules with ontology-guided LLM processing and evidence-based grounding.
3 System Architecture
- The pipeline identifies missing or outdated facts, retrieves documents, applies pattern-based and LLM-based extractors, then normalizes, ranks, and consolidates candidates.Figure 1 depicts the system architecture and its LLM-based extractor.Source: §3 · p. 2, Figure 1 · p. 3
3.1 Extraction Initiator
- The Extraction Initiator uses web-page activity to schedule linked entities for reevaluation, avoiding exhaustive reprocessing of all KG subjects.In the described time-lag analysis, Wikipedia facts appear an average of 69 days before Wikidata; outputs are subject, URL, and locale tuples.Source: §3.1 · p. 3
3.2 Evidence Retriever
- Under a controlled-domain assumption, the Evidence Retriever fetches requested documents from a web-crawl index, primarily using Wikipedia and deterministic, language-specific entity-to-URL mappings.The cited example maps an entity to canonical Wikipedia URLs across locales.Source: §3.2 · p. 3
3.3 Knowledge Extractor
- ODKE+ combines pattern-based extraction for semi-structured sources with LLM-based extraction that generalizes across contexts and data formats.Pattern rules map infobox keys to predicates, extract values, aggregate candidates, and validate results against KG type constraints.Source: §3.3 · pp. 3–4
- For LLM extraction, type-specific ontology snippets supply applicable predicates and descriptions, qualifiers, and acceptable units to guide structured, schema-aligned outputs.The LLM extracts key-value pairs from context, maps values to ontology predicates, and serializes results for downstream normalization and ingestion.Source: §3.3 · p. 4
- Snippet generation ranks predicates by frequency and semantic importance, enriches them with ontology, Wikidata, and KG information, and configures normalization rules and expected units.The resulting token-efficient schema fragments are periodically refreshed and support schema-aware extraction across diverse entity types and structured or unstructured content.Source: §3.3 · pp. 4–5
3.3.1 Grounder
- The Grounder turns each extracted triple into a natural-language assertion and retains it only when a second LLM judges it supported by the evidence context.The paper describes this as adding factual control without significant latency or computational cost.Source: §3.3.1 · p. 5
3.4 Corroborator
The Corroborator standardizes candidate facts, consolidates equivalent values, and ranks them to select trustworthy values for each predicate.
3.4 Corroborator
- For some extractions, fallback normalization converts values such as “6 ft” or “July 4, 1776” into standardized numeric or temporal forms.Predicate-specific normalization logic is applied when configured.Source: §3.4 · p. 5
- The Corroborator groups facts by subject and predicate, consolidates equivalent values, removes inconsistencies, and scores candidates using extractor type, confidence, source frequency, and answer richness.Answer richness includes supporting qualifiers for the main fact.Source: §3.4 · p. 5
- Low-ambiguity predicates may use rule-based ranking, while complex cases can use AutoML-learned weighting functions to rank facts.The paper names H2O as an example package for AutoML-based scoring.Source: §3.4 · p. 5
3.5 Data Export & KG Ingestion
ODKE+ exports validated, ranked facts as normalized triples with optional qualifiers and provenance or confidence metadata. Ingestion aligns them with the canonical ontology and links entities to existing KG nodes or creates new ones when needed.
3.5 Data Export & KG Ingestion
- Exported facts are high-confidence, normalized subject-predicate-object triples that may include qualifiers, provenance, and confidence scores.These triples enter the KG as a new data source.Source: §3.5 · p. 5
- The source-ingestion pipeline aligns exported triples with the canonical ontology.Source: §3.5 · p. 5
- Entities with globally unique identifiers are linked directly to existing KG nodes; otherwise, machine-learning entity linking checks for a match.The text names Wikidata QIDs and Wikipedia URLs as examples of globally unique identifiers.Source: §3.5 · pp. 5–6
- When entity linking finds no match, ODKE+ creates a new entity and inserts it with the triple, preserving it with minimal assumptions.This hybrid strategy is described as supporting semantic accuracy and incremental extension.Source: §3.5 · p. 6
3.6 Deployment and Continuous Update
ODKE+ supports periodic batch updates and real-time streaming ingestion with strict schema validation. Its monitoring includes weekly triple audits and side-by-side user-query evaluations.
3.6 Deployment and Continuous Update
- ODKE+ supports periodic large-scale batch updates and real-time ingestion of ongoing updates, with strict schema validation for ontology consistency.Source: §3.6 · p. 6
3.7 Monitoring, QA, and Feedback Loops
- Weekly audits sample approximately 2,000 triples, and ODKE+ has consistently met the ≥95% precision threshold required to remain in production.Source: §3.7 · p. 6
- In weekly side-by-side evaluations of real user KG queries, annotators prefer ODKE+ answers in roughly two-thirds of cases.The answers are shown anonymously alongside the incumbent answers.Source: §3.7 · p. 6
4 Results and Impact
ODKE+ reports production-scale streaming, high precision, broader query coverage, and earlier fact availability. Component evaluations also report gains from grounding and corroboration.
4 Results and Impact
- 150–250 K new facts per day are processed in streaming mode for high-priority entities, with end-to-end latency under 2 hours for most updates.The system has been deployed in production since May 2025 and runs batch and near-real-time streaming updates.Source: §4 · p. 6
4.1 Quality Evaluation and Component Impact
- Weekly audits report over 95% factual accuracy, query coverage is 10% higher than prior pipelines, and newly extracted facts appear roughly 50 days earlier than in legacy workflows.Source: §4.1 · p. 6
- Declarative predicate integration, schema validation, and retry mechanisms keep success rates above 99.9%.Source: §4.1 · p. 6
- 35% fewer hallucinated extractions followed grounding, while corroboration raised factual precision from 91% for raw LLM output to 98.8% post-ranking.Source: §4.1 · p. 6
4.2 Production Metrics
- Table 2 summarizes key performance indicators from the latest deployment snapshot, covering scale across diverse predicates and entity types alongside factual precision.The supplied table passage identifies the table but provides no individual metric values.Source: §4.2 · p. 6, Table 2 · p. 6
5 Conclusion
ODKE+ is presented as a production-grade system for ontology-guided open-domain extraction, with a lightweight grounding verifier. Its deployment spans millions of facts across 195 predicates, and the authors describe it as demonstrating the viability of LLM-based extraction in real-world KG pipelines.
5 Conclusion
- ODKE+ combines ontology-guided prompting and ontology-oriented extraction at scale with a lightweight grounding verifier.Source: §5 · p. 6
- The deployed system extracts and ingests millions of facts across 195 predicates, with support for batch and real-time use cases.Source: §5 · p. 6
- The authors conclude that ODKE+ demonstrates the viability of LLM-based extraction in real-world KG pipelines and offers a blueprint for trustworthy, ontology-aligned extraction at scale.Source: §5 · p. 6
Ethics Statement
ODKE+ discusses ethical considerations involving bias, privacy, verification, and environmental impact, describing both safeguards and unresolved challenges.
Ethics Statement
- Underlying social or cultural biases in broad-web LLM training data may propagate through ODKE+, and mitigating them remains future work.Grounding and corroboration target factual accuracy but do not resolve underlying model biases.Source: Ethics Statement · p. 7
- ODKE+ processes public sources such as Wikipedia, extracts explicitly stated facts, and avoids user-generated or private data.Its architecture is designed to avoid suppositional inference and hallucinated facts.Source: Ethics Statement · p. 7
- A second-LLM Grounder checks context alignment, while a Corroborator filters and scores extractions to mitigate hallucination risks.The paper reports that these design choices improved factual precision in production.Source: Ethics Statement · p. 7
- ODKE+ uses lightweight pattern-based extractors where possible and applies LLMs selectively based on data complexity to address computational cost.Batch processing and pipeline orchestration are also optimized for cost and efficiency.Source: Ethics Statement · p. 7
A Sample Prompt for LLM-based Extraction
The sample extraction prompt pairs page content with an ontology snippet, instructing the LLM to return schema-aligned facts as valid JSON without inference.
A Sample Prompt for LLM-based Extraction
- The example prompt supplies the LLM with an entity-specific ontology snippet and web-page content to guide fact extraction.It frames the task as mapping passage facts using the provided ontology.Source: A Sample Prompt for LLM-based Extraction · p. 8
- The output must be a valid JSON dictionary keyed by ontology properties with supporting mentions, and each property's value is a list of objects.Each object requires an answer field and may include qualifiers, a normalized answer, and a normalization unit.Source: A Sample Prompt for LLM-based Extraction · p. 8
- The prompt restricts extraction to facts explicitly stated in the passage, forbids inference or retrieval from memory, and requires syntactically correct JSON.Source: A Sample Prompt for LLM-based Extraction · p. 8
B Sample Prompt for LLM as a judge (Grounder)
The Grounder prompt asks a second LLM to check whether a candidate triple is supported by context, using a True-or-False judgment.
B Sample Prompt for LLM as a judge (Grounder)
- The Grounder receives a subject, predicate with optional qualifier, object, and context, then returns True or False for whether the triple is grounded.The prompt notes that an object may appear in the context as part of a list.Source: B Sample Prompt for LLM as a judge… · p. 8
- The example pairs Felton Ross biographical context with a candidate date-of-birth triple for verification.The context includes his birth, death, nationality, education, occupation, spouse, children, and research information.Source: B Sample Prompt for LLM as a judge… · p. 8
- The accompanying ontology gives property descriptions and qualifier information, including normalization guidance for height in centimetres.Examples cover birth and death dates, occupation, education, spouse, child, and height properties.Source: B Sample Prompt for LLM as a judge… · p. 8