Source-linked AI summary
KGGen: Extracting Knowledge Graphs from Plain Text with Language Models
Belinda Mo, Kyssen Yu, Joshua Kazdan, Joan Cabezas, Proud Mpala, Lisa Yu, Chris Cundy, Charilaos Kanatsoulis, Sanmi Koyejo
TL;DR
Knowledge graphs are scarce, incomplete, and often difficult to use because automatically extracted graphs can be sparse and poorly normalized. KGGen combines language-model extraction with clustering-based entity and edge resolution, and introduces MINE benchmarks for evaluation. It performs comparably to GraphRAG while scaling better as source text grows.
Problem
Scarce, incomplete, and insufficiently verified knowledge-graph data limits downstream applications such as graph RAG, KG embeddings, and synthetic graph training data.
Method
KGGen uses language models to extract triples, then aggregates and iteratively clusters and resolves duplicate entities and edges.
Results
KGGen performs comparably to GraphRAG on the proposed benchmarks and scales better in information compression and graph sparsity as source text grows.
Takeaways & Limitations
KGGen provides an open-source Python package and benchmarks for generating and evaluating dense knowledge graphs from plain text.
Takeaways & Limitations
The benchmarks cover corpora up to 5M tokens, below the web-scale text needed for a knowledge-graph foundation model.
Abstract
from arXiv · showhide
Recent interest in building foundation models for KGs has highlighted a fundamental challenge: knowledge-graph data is relatively scarce. The best-known KGs are primarily human-labeled, created by pattern-matching, or extracted using early NLP techniques. While human-generated KGs are in short supply, automatically extracted KGs are of questionable quality. We present a solution to this data scarcity problem in the form of a text-to-KG generator (KGGen), a package that uses language models to create high-quality graphs from plaintext. Unlike other KG extractors, KGGen clusters related entities to reduce sparsity in extracted KGs. KGGen is available as a Python library (\texttt{pip install kg-gen}), making it accessible to everyone. Along with KGGen, we release the first benchmark, Measure of of Information in Nodes and Edges (MINE), that tests an extractor's ability to produce a useful KG from plain text. We benchmark our new tool against existing extractors and demonstrate far superior performance.
1 Introduction
KGGen addresses scarce, incomplete, and sparse knowledge-graph data by extracting and consolidating graphs from plain text. The paper also introduces benchmarks for measuring extraction quality and reports stronger scaling with increasing source-text size.
- KGGen: KGGen uses language models to extract subject-predicate-object triples and clusters equivalent entities and edges to produce denser graphs.The clustering process identifies nodes referring to the same underlying entities and consolidates edges with equivalent meanings.
- Benchmarks: The paper introduces two benchmarks that measure information retention and knowledge retrieval for graphs generated from plain text.The benchmarks target short texts and multi-million-token, web-based knowledge databases.
- Results: KGGen performs comparably to GraphRAG on the proposed benchmarks while scaling better in information compression and graph sparsity as databases grow.The comparison is reported for increasing plain-text database length.
- Contributions: The authors release KGGen as an open-source Python package for extracting high-quality knowledge graphs from plain text.The package is intended to make the method broadly accessible.
- Contributions: The paper develops benchmarks to measure KGGen’s performance and reports improved scaling relative to past methods as the text source grows.These contributions target progress measurement and larger source collections.
2 Related Work
Related work spans rule-based, neural, and language-model-powered knowledge-graph extraction, alongside evolving evaluation methods. Recent evaluation increasingly emphasizes semantic coherence, completeness, consistency, and downstream usability.
- KG Extraction: YAGO exemplifies early rule-based extraction, while OpenIE represents later neural and natural-language-inference-based extraction from text.These approaches illustrate the field’s progression from hard-coded rules toward more advanced NLP systems.
- KG Extraction: Transformer-based methods now extract complex entity relationships through fully automatic and human-assisted approaches.The paper positions KGGen within this newer extraction literature.
- KG Extraction: KGGen’s extraction contribution is to build graphs conducive to embedding algorithms such as TransE and TransR.The supplied passage frames this as the paper’s contribution to extraction research.
- KG Evaluation: Knowledge-graph evaluation has expanded from direct completeness and connectivity assessment toward semantic coherence and downstream usability.This shift reflects increasing attention to how graphs function in applications.
- KG Evaluation: Earlier evaluation used expert annotations for correctness, but this approach was laborious and prone to errors as graphs grew larger and more diverse.Subsequent work introduced approximation and interactive annotation methods, alongside metrics such as completeness.
- KG Evaluation: LP-Measure evaluates graph quality by removing triples and testing whether link-prediction tools can recover them without human labor or a gold standard.The method assesses consistency and redundancy and can distinguish between good and bad knowledge graphs.
3 Existing Methods
Existing plain-text-to-KG methods include OpenIE and GraphRAG, which extract entities and relations using different NLP and language-model pipelines. These methods provide the main comparison basis for KGGen.
- Comparison Basis: The paper uses OpenIE and GraphRAG as bases for comparison with KGGen’s plain-text knowledge-graph extraction approach.Both are presented as prominent existing synthetic extractors.
- OpenIE: OpenIE parses sentence dependencies, splits complex sentences into self-contained clauses, and produces subject-relation-object tuples.A trained classifier decides how to traverse dependency-parse edges while processing clauses.
- GraphRAG: GraphRAG prompts language models to extract entities and relationships that form a graph for later graph-based retrieval.Few-shot examples guide the language model toward desirable extractions.
4 KGGen: Knowledge Graphs From Plain Text
KGGen uses a multi-stage pipeline that extracts graphs from text, aggregates them, and resolves duplicate entities and edges. Its hybrid clustering and language-model process is designed to scale de-duplication across large graphs while preserving distinctions between related concepts.
- Pipeline: KGGen extracts entities and relations from each source text, aggregates graphs across sources, and iteratively resolves duplicate entities and edges.The pipeline combines language-model extraction with traditional information-retrieval methods during resolution.
- Pipeline: Prompting constrains the language model against conflating distinct concepts such as Type 1 diabetes and Type 2 diabetes.The same constraint applies to potentially similar but nonidentical entities and edges.
- Pipeline: Entity and edge resolution prevents sparse graphs that may produce meaningless embeddings under algorithms such as TransE.The paper links resolution to reducing sparsity in downstream graph representations.
- Extraction: The first stage converts unstructured text into triples by separately extracting entities and then subject-predicate-object relations with a language model.The implementation uses Gemini 2.0 Flash and structured DSPy outputs.
- Aggregation: Aggregation combines unique entities and edges across source graphs, lowercases them, and reduces redundancy without using a language model.This stage produces a single combined graph before resolution.
- Entity and Edge Resolution: Resolution merges nodes and edges representing the same real-world entity or concept through embedding-based clustering followed by language-model de-duplication.The two-stage process is applied separately to entity and edge items.
- Entity and Edge Resolution: The resolution loop retrieves semantically similar items, identifies exact duplicates, selects canonical representatives, and repeats until each cluster is exhausted.The process accounts for tense, plurality, case, abbreviations, and shorthand forms.
- Entity and Edge Resolution: On the largest 20M chunk dataset, the method consolidated variants such as Olympic Winter Games, Winter Olympics, and winter Olympic games.The example demonstrates parallel semantic-cluster processing on a large graph.
5 Benchmarks for Extraction Performance
MINE introduces benchmarks for measuring how well plain-text-to-KG extractors capture information and support retrieval. MINE-1 tests knowledge retention from short texts, while MINE-2 evaluates downstream RAG performance on large datasets.
- MINE is introduced as a benchmark for measuring an extractor’s ability to capture and distill text into a knowledge graph.
- 5.1 MINE-1: Knowledge Retention: MINE-1 evaluates whether a graph preserves information from short, page-length articles without relying on downstream tasks.The benchmark uses articles paired with verified facts and checks whether those facts can be recovered from extracted graphs.
- 5.2 MINE-2: Retrieval-Augmented Generation: MINE-2 measures downstream retrieval-augmented generation performance on multi-million-token datasets using WikiQA questions and aggregated knowledge graphs.Questions and graph triples are embedded for retrieval, and an LLM synthesizes and judges answers using retrieved triples and their source text chunks.
- 5.1 MINE-1: Knowledge Retention: MINE-1 contains 100 articles, each paired with 15 verified facts, covering Arts, Culture & Society, Science, Technology, Psychology/Human Experience, and History & Civilization.Articles average 592 words, with a standard deviation of 85 words and a range of 440–976 words.
- 5.1 MINE-1: Knowledge Retention: MINE-1 scores each extractor by the percentage of facts recovered from its graph, averaged across all 100 articles.Fact recovery uses semantic retrieval, two-relation subgraph expansion, and an LLM binary judgment; manual validation achieved 90.2% agreement and 0.80 correlation with LLM judgments.
6 Results
KGGen is evaluated against OpenIE and GraphRAG on extraction fidelity, retrieval, entity coverage, graph structure, relation reuse, scalability, and cost. It generally performs strongly, while matching GraphRAG on MINE-2 and preserving performance across language models.
- MINE-1 evaluation: KGGen outperforms OpenIE and GraphRAG on MINE-1 extraction fidelity across 100 articles.The benchmark compares the three extractors using MINE-1 accuracies and example relations.
- Generalization across language models: 73% was KGGen’s highest MINE-1 score, achieved with Claude Sonnet 3.5 across tested language models.The evaluation reports strong performance across multiple state-of-the-art LLMs.
- Triple validity: 98% of KGGen’s sampled triples were valid, compared with 55% for OpenIE and 0% for GraphRAG.Validity was manually assessed on 100 randomly selected triples from each method.
- Entity extraction: KGGen captured both human-annotated target entities in 96% of SemEval-2010 Task 8 sentences.KGGen often extracted more specific entity descriptions and additional relevant entities than the annotations.
7 Broader Impact and Community Adoption
KGGen addresses KG scarcity through an open-source plain-text extractor and has attracted substantial community adoption. The package has received over 700 GitHub stars and more than 12,000 downloads.
- Contribution and adoption: KGGen is an open-source plain-text-to-KG extractor distributed as a Python package.The work presents the package as a way to help address knowledge-graph scarcity.
- Contribution and adoption: The package has received over 700 GitHub stars and more than 12,000 downloads since release.These figures quantify reported community adoption.
8 Limitations and Future Work
KGGen still faces entity and relation de-duplication errors, while its benchmarks remain limited to corpora of up to 5M tokens and domain-specific extraction poses additional challenges.
- KGGen can over- or under-deduplicate entities and relations, motivating further research into entity resolution.
- Its benchmarks cover corpora of up to 5M tokens, which does not represent web-scale text needed for a KG foundation model.
- Future benchmark expansions could use larger corpora to better measure the practicality of extraction techniques.
- Medicine and finance require specialized domain knowledge that general-purpose LLMs may lack, potentially limiting extraction quality relative to human experts.
- Incorporating domain-specific ontologies could improve extraction precision, while adaptive ontology integration remains a proposed direction.
A Prompts for KG Extraction
The appendix documents prompts for KG extraction and de-duplication, then specifies the prompts MINE uses to generate facts, evaluate retrieval, answer questions, and check answer containment.
- A Prompts for KG Extraction: The appendix states that it provides the exact prompts used to extract KGs.
- A Prompts for KG Extraction: Initial KG extraction uses separate prompts for entities and subject-predicate-object relations passed as DSPy signature descriptions.
- A Prompts for KG Extraction: The entity prompt requests thorough, accurate extraction of key subjects and objects from the source text.
- A Prompts for KG Extraction: The relation prompt restricts subjects and objects to the previously extracted entity list and requests faithful triples.
- A Prompts for KG Extraction: After extraction, de-duplication uses prompts for entity or edge resolution.
- A Prompts for KG Extraction: The resolution prompt identifies duplicates by equivalent meaning and returns an alias representing them, or an empty list when none exist.
- B Prompts for MINE: MINE prompts an LLM to extract 15 directly supported facts from an article, then evaluates whether query context contains each correct answer.
- B Prompts for MINE: For RAG evaluation, MINE supplies graph triples and text evidence to answer questions, then checks whether responses contain expected WikiQA answers.
C Example Article from MINE
The MINE appendix example is an article titled “The Rise of Cryptocurrencies,” describing cryptocurrencies’ rapid financial impact and continuing adoption despite uncertainty.
- The example article is titled “The Rise of Cryptocurrencies.”
- It describes cryptocurrencies as transforming views of money and transactions while becoming significant in the global economy.
- The article states that some countries embrace cryptocurrencies and blockchain for innovation and economic growth, while others remain cautious.
- It attributes caution to concerns including money laundering, tax evasion, and consumer protection.
- The article concludes that adoption by individuals and businesses is rapidly changing finance despite ongoing challenges and uncertainties.
D Additional Graphs
Figure 7 compares relation-type reuse as graph size increases: KGGen tends to reuse relation types more frequently, whereas GraphRAG remains near two instances per type.
- As graphs grow, KGGen tends to reuse each unique relation type more frequently.
- GraphRAG maintains an average usage of about 2 instances per relation type regardless of graph size.
NeurIPS Paper Checklist
The paper’s checklist responses state that its abstract and introduction accurately reflect its contributions, that reproducibility information is provided, and that it includes a limitations section. It reports no theoretical claims and states that statistical error bars are reported where possible.
- The abstract and introduction accurately reflect the paper’s contributions and scope.
- The paper includes a dedicated limitations section and reports no theoretical claims requiring assumptions or proofs.
- The paper states that it provides sufficient methodological and experimental details to reproduce its main results.It also reports plans to release the code publicly and provide code in supplementary material.
- The paper reports statistical error bars where possible for its experimental results.The checklist guidance states that error bars should identify the variability captured and whether they represent standard deviation or standard error.
- The experiments do not require special hardware and can run on most laptops with production models, requiring only a model-provider API key.