PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoSchemaKG: Autonomous Knowledge Graph Construction through Dynamic Schema Induction from Web-Scale Corpora

Jiaxin Bai, Wei Fan, Qi Hu, Qing Zong, Chunyang Li, Hong Ting Tsang, Hongyu Luo, Yauwai Yim, Haoyu Huang, Xiao Zhou, Feng Qin, Tianshi Zheng, Xi Peng, Xin Yao, Huiwen Yang, Leijie Wu, Yi Ji, Gong Zhang, Renhai Chen, Yangqiu Song

arXiv:2505.23628v3cs.CLcs.AI

TL;DR

Knowledge graphs support applications such as search and question answering, but their construction often depends on expert-defined schemas that limit scalability, adaptability, and domain coverage. AutoSchemaKG extracts entity and event triples while inducing conceptual schemas from text, producing billion-scale ATLAS graphs. It reports gains on multi-hop question answering and LLM factuality, alongside high alignment with human-crafted schemas without manual intervention.

  • Problem

    Knowledge graphs support applications including search and question answering, but expert-defined schemas constrain their scalability, adaptability, and domain coverage.

    Source: §1 · p. 1

  • Method

    AutoSchemaKG uses LLMs to extract triples involving entities and events while dynamically inducing schemas through conceptualization, without predefined schemas.

    Source: §1 · pp. 1–2, §3.1 · p. 3

  • Results

    AutoSchemaKG outperforms state-of-the-art baselines by 12-18% on multi-hop question answering tasks and enhances LLM factuality by up to 9%; ATLAS contains 900+ million nodes and 5.9 billion edges.

    Source: §1 · p. 3, §7 · p. 9

  • Takeaways & Limitations

    Billion-scale knowledge graphs with dynamically induced schemas can complement parametric knowledge in large language models.

    Source: §7 · p. 9

    Constructing billion-scale knowledge graphs requires 78,400+ GPU hours, limiting accessibility for researchers with resource constraints.

    Source: §8 · p. 9

Abstract

from arXiv · show

We present AutoSchemaKG, a framework for fully autonomous knowledge graph construction that eliminates the need for predefined schemas. Our system leverages large language models to simultaneously extract knowledge triples and induce comprehensive schemas directly from text, modeling both entities and events while employing conceptualization to organize instances into semantic categories. Processing over 50 million documents, we construct ATLAS (Automated Triple Linking And Schema induction), a family of knowledge graphs with 900+ million nodes and 5.9 billion edges. This approach outperforms state-of-the-art baselines on multi-hop QA tasks and enhances LLM factuality. Notably, our schema induction achieves 92\% semantic alignment with human-crafted schemas with zero manual intervention, demonstrating that billion-scale knowledge graphs with dynamically induced schemas can effectively complement parametric knowledge in large language models.

1 Introduction

AutoSchemaKG addresses the dependence on expert-defined schemas that limits knowledge-graph scalability, adaptability, and domain coverage. It extracts entity and event triples while inducing schemas from text, then evaluates the resulting graphs on downstream tasks and schema alignment.

1 Introduction

  • AutoSchemaKG simultaneously extracts triples and dynamically induces schemas from text without predefined schemas, modeling events alongside entities.Its extraction framework includes entity relationships, event structures, and their conceptual categorizations.Source: §1 · pp. 1, 3
  • With event information, AutoSchemaKG preserves over 90% of original passage content versus just 70% for entity information alone.The paper links event modeling to capturing temporal and causal relationships that entity-only graphs miss.Source: §1 · p. 2
  • Conceptualization generalizes specific entities, events, and relations into broader categories, creating semantic bridges and supporting zero-shot inference across domains.The paper describes the resulting conceptualization layer as transforming disconnected facts into a coherent knowledge ecosystem with emergent reasoning capabilities.Source: §1 · p. 2
  • Applying the framework to three Dolma 1.7 subsets produced ATLAS graphs with over 900 million nodes and 5.9 billion edges.The subsets are English Wikipedia, Semantic Scholar paper abstracts, and 3% of Common Crawl data.Source: §1 · p. 2
  • 12-18% on multi-hop question answering tasks and up to 9% on LLM factuality; schema induction achieves 92% semantic alignment with human-crafted schemas with zero manual intervention.The reported downstream gains are attributed to integrating entities, events, and conceptual abstractions.Source: §1 · pp. 2–3
  • The ATLAS-family graphs process web-scale data into billions of triples, and their RAG pipeline is reported effective across diverse domains without domain-specific customization.The authors describe the graphs as, to their knowledge, the largest automatically constructed knowledge graphs and graph-based RAG datasets available.Source: §1 · p. 3

2 Problem Definition

The section defines a knowledge graph with a conceptual schema that represents entities and events, their relations, and the concepts assigned to them.

2 Problem Definition

  • A knowledge graph with a conceptual schema contains disjoint entity and event nodes, relation-typed edges, and conceptual categories.Each node and relation type is assigned a nonempty subset of concepts.Source: §2 · p. 3
  • Edges may connect entity–entity, entity–event, or event–event nodes.The graph’s node set separates event nodes from entity nodes.Source: §2 · p. 3
  • The schema maps each node and relation type to one or more conceptual categories.The mappings are represented by ϕ for nodes and ψ for relation types.Source: §2 · p. 3

3 AutoSchemaKG Framework

AutoSchemaKG uses a multi-stage LLM pipeline to extract entity and event relationships, then induces conceptual types for graph elements without manual curation.

3 AutoSchemaKG Framework

  • The framework automates knowledge graph construction through LLM-based conversion of unstructured text into triples.The pipeline is applied to the Dolma corpus.Source: §3 · p. 3, §3.1 · p. 3

3.1 Triple Extraction

  • The triple-extraction pipeline builds entity–entity, entity–event, and event–event relationships in three sequential LLM stages.Texts are filtered for English and segmented when they exceed token limits before batched processing.Source: §3.1 · p. 3

3.2 Schema Induction

  • Schema induction generates conceptual phrases for entities, events, and relations, mapping graph elements to concept sets without manual curation.For entities, neighboring-node context supplies additional semantic cues; the LLM generates at least three phrases per element at varying abstraction levels.Source: §3.2 · p. 4

4 Construction of ATLAS Families

The ATLAS family is constructed from three Dolma corpus sources, with substantial GPU compute used to extract billions of semantic relationships.

4 Construction of ATLAS Families

  • ATLAS-Wiki uses Wikipedia and Wikibooks, ATLAS-Pes2o uses Semantic Scholar abstracts, and ATLAS-CC uses 3% samples from Common Crawl’s head, middle, and tail.The Common Crawl segments are selected to measure distribution similarity to Wikipedia text.Source: §4 · p. 4
  • Construction used approximately 78,400 GPU hours across En-Wiki, Pes2o-Abstract, and Common Crawl graphs.The reported workloads were 14,300, 11,800, and 52,300 GPU hours, respectively.Source: §4 · p. 4
  • The graphs were built with Llama-3-8B-instruct on 80GB GPUs using 1024-token chunks in batches.The reported setup used 1,513 TFLOPS of FP16 compute and Flash Attention.Source: §4 · p. 4

5 Experiment

Experiments evaluate extraction, schema induction, and information preservation, then test ATLAS in multi-hop QA, factuality, and knowledge-intensive reasoning. Results show gains across these tasks, alongside stronger preservation from event-level representations and domain-specific performance patterns.

5.1 Evaluating AutoSchemaKG

  • Triple precision, recall, and F1 exceed 90% in most cases, with particularly strong extraction performance on Wikipedia.The evaluation used DeepSeek-V3 to identify false positives and missing facts.Source: §5.1 · p. 5, Table 2 · p. 5
  • Event and combined representations yield MCQ performance closer to the full-passage upper bound than entity-only triples.The reported entity-only accuracy exceeds 95% in most cases.Source: §5.1 · p. 5, Table 3 · p. 5
  • Schema induction achieves over 80% recall, often 90%, for entity, event, and relation types, with performance improving as LLM size increases.The evaluation reports results across three LLaMA model sizes.Source: Table 4 · p. 5, §5.1 · p. 6

5.2 Performance on Multi-hop QA Tasks

  • Event nodes help retrieve relevant subgraphs when triple information is ambiguous or missed, while concept nodes link otherwise disparate subgraphs for multi-hop QA.The described concept-node pathway uses PageRank to bridge separate subgraph influences.Source: §5.2 · p. 7

5.3 Enhancing LLM Factuality with KGs

  • HippoRAG2 with the KG outperforms baselines on Wikipedia (56.43% accuracy, 30.48% F1) and Common Crawl, with competitive results on Pes2o-Abstract.The evaluation used FELM and LLaMA-3.1-8B-Instruct; Wikipedia performance may reflect FELM samples being partially sourced from Wikipedia.Source: Table 6 · p. 7, §5.3 · p. 8

5.4 General Domain Knowledge Capabilities

  • 4-point improvement over baseline in Law; ATLAS variants enhanced performance across tested knowledge-intensive MMLU domains, while ToG on ATLAS performed best in Logic.ATLAS-CC performed best in Law and History; ATLAS-Pes2o and ATLAS-Wiki showed strengths in other subject areas.Source: §5.4 · pp. 8–9, Table 7 · p. 8
  • Knowledge graphs show limited benefits in mathematical and technical domains, where node-relation structures are less effective for procedural knowledge.The reported domain-specific pattern favors factual relationships in humanities and social sciences.Source: §5.4 · p. 9

6 Further Related Work

Prior work spans automated concept harvesting, supervised concept prediction, LLM-based ontology learning, and varied schema-induction and text-to-graph methods.

6 Further Related Work

  • Hearst (1998) used lexico-syntactic patterns to harvest concepts, and Agirre et al. (2000) extracted topically related concepts from web documents.These approaches also enhanced WordNet.Source: §6 · p. 9
  • Recent related systems include SAC-KG for specialized KGs, Docs2KG for heterogeneous documents, and KAG for KG-text indexing in multi-hop reasoning.Other work surveys ontology-learning pipelines and addresses canonicalization or bidirectional text-to-graph conversion.Source: §6 · p. 9
  • Txt2onto used text representations and a supervised classifier to predict terms and concepts, while LLMs4OL applied zero-shot LLM prompting to ontology learning.The LLMs4OL discussion continues in p17.Source: §6 · p. 9

7 Conclusion

AutoSchemaKG constructs large knowledge graphs using LLM-based triple extraction and schema induction without predefined schemas. The paper reports improved downstream performance and high schema alignment without manual intervention.

7 Conclusion

  • AutoSchemaKG constructs the ATLAS family with 900+ million nodes and 5.9 billion edges, reporting improved multi-hop QA and LLM factuality.The conclusion describes triple extraction as having >95% precision and says schema induction is performed without expert intervention.Source: §7 · p. 9

8 Limitations

The paper identifies resource demands, inherited LLM biases, difficulty with extremely technical domains, and possible graph inconsistencies or gaps as limitations.

8 Limitations

  • 78,400+ GPU hours limit accessibility for resource-constrained researchers, while LLM limitations may affect specialized domains and extremely technical domains remain difficult to organize.The authors also note possible inconsistencies, contradictions, or information gaps in sparse knowledge regions.Source: §8 · p. 9

9 Ethical Statement

The ethical statement describes use of public corpora, attribution, privacy practices, and disclosed computational requirements. The included implementation passages detail staged triple extraction and schema induction, with contextual enhancement applied to entity abstraction.

9 Ethical Statement

  • The work uses publicly available Dolma 1.7 data with attribution, processes no personally identifiable information, and discloses approximately 78,400 GPU hours.It acknowledges possible source-text and LLM biases and presents methodological detail to support reproducibility and validation.Source: §9 · pp. 9–10

A Prompts for Triple Extractions

B Implementation Details of Knowledge Graph Construction Framework

  • The construction framework uses LLMs to extract three relationship categories in sequential stages, with batch processing, text segmentation, and structured JSON outputs.It is designed for large-scale processing and supports different model and dataset constraints through parameters such as Lmax, Cmax, and B.Source: B Implementation Details of Knowledge… · p. 14, §B.1 · p. 14, §B.1.5 · p. 15

B.1.1 Text Preprocessing and Data Organization

  • English texts are filtered or assumed when metadata is unavailable, then long documents are chunked within the model’s token capacity and grouped into batches.Chunks retain identifiers and metadata to preserve traceability and contextual integrity.Source: §B.1.1 · p. 14

B.1 Triple Extraction

  • The three extraction stages produce entity-entity triples, bidirectional entity-event triples, and event-event triples for causal, temporal, or logical connections.Entity-entity outputs are parsed as (e1, r, e2), entity-event outputs link entities and events in either direction, and event descriptions may use an extended generation limit.Source: §B.1.2 · p. 15, §B.1.3 · p. 15, §B.1.4 · p. 15

B.2 Schema Induction

  • Schema induction abstracts events, entities, and relations into conceptual phrases that form concept set C and map graph elements to concepts.The method processes elements in batches and targets at least three short phrases per element, varying abstraction level and avoiding repetition or the input term.Source: §B.2 · pp. 15–16, §B.2.1 · p. 16

B.2.2 Contextual Enhancement for Entities

  • Entity abstraction samples neighboring nodes and their relations to give the LLM graph context, while event and relation abstractions rely only on textual descriptions.For example, the context “started by John Andrew Rice” helps abstract Black Mountain College as a college, school, or liberal arts college.Source: §B.2.2 · pp. 16–17, Figure 5 · p. 16, Figure 6 · p. 17

B.2.3 Implementation Details

Schema induction batches the extracted graph for LLM processing, optionally sampling batches to reduce experimental processing time. The resulting concepts map nodes and relations to broader types, supporting adaptation across domains without manual curation.

B.2.3 Implementation Details

  • The pipeline partitions the serialized graph into batches and can process selected slices or randomly sample batches to reduce experimental processing time.The graph is typically stored in binary format and loaded into memory.Source: §B.2.3 · p. 17
  • Figure 7 presents prompts for conceptualizing relations with added context.Source: Figure 7 · p. 17
  • GPU-based batched LLM inference generates abstracted phrases, which are saved with each element’s type and aggregated into unique concepts and coverage statistics.The implementation uses a model-specific chat template and records whether each element is an event, entity, or relation.Source: §B.2.3 · p. 17
  • Mapping each node and relation to concepts abstracts instances into general types, making the schema adaptable across domains without manual curation.The mappings are represented by ϕ for nodes and ψ for relations.Source: §B.2.3 · p. 17

C.1 Datasets

This block describes datasets and metrics for evaluating entity, event, and relation induction, then outlines graph-based retrieval and downstream evaluations. It also reports case-study examples and retrieval challenges across multi-hop QA datasets.

C.1 Datasets

  • Entity induction is evaluated on FB15kET and YAGO43kET test entities, event induction on wikiHow test events, and relation induction on FB15kET test relations.The datasets provide ground-truth types; the relation types are derived from FB15kET domain segments.Source: §C.1 · p. 18

C.2 Metrics

  • BertScore, BS-R, and BS-C compare induced schemas with ground-truth types and measure how many types are recalled per instance or across the test set.The token embeddings for ground-truth and induced types are obtained with RoBERTa.Source: §C.2 · p. 18

D Case Study Examples

  • Event nodes add contextual information and concept nodes bridge otherwise disconnected subgraphs, illustrating complementary pathways for graph-based reasoning.Figure 9 contrasts event-enriched context with triplets, while Figure 10 illustrates concept-node pathways.Source: D Case Study Examples · p. 18

E Algorithm for RAG

  • The RAG implementations adapt Think-on-Graph and HippoRAG2 to entity-event-concept graphs, with variants for focused traversal and large-scale exploration.The large-scale variant uses dynamic graph sampling and common-word filtering.Source: E Algorithm for RAG · pp. 18, 20, 21

F The Recall Metrics in Opendomain QA Tasks

G.1 Implementation and Evaluation Details on FELM

  • FELM factuality evaluation uses balanced accuracy and F1 to assess detection of false segments across three domains, while math and reasoning use the vanilla setting.The three domains are world knowledge, science and technology, and writing/recommendation.Source: §G.1 · p. 22

G.2 Implementation and Evaluation Details on MMLU

On MMLU, retrieval-augmented generation improves performance in knowledge-intensive domains but can reduce performance on reasoning-focused tasks. The evaluation organizes subjects into categories and compares graph-based methods with retrieval baselines across corpora.

G.2 Implementation and Evaluation Details on MMLU

  • Table 11 maps subjects to tasks to support domain-specific performance analysis.Source: Table 11 · p. 24
  • Retrieval-augmented generation enhances knowledge-intensive MMLU performance but can hurt mathematics and logical-reasoning performance.Source: §G.2 · p. 22, Table 12 · p. 24
  • Table 12 compares KG-integrated HippoRAG2 and ToG with Random, BM25, and Dense Retrieval across Wikipedia, Common Crawl, and Pes2o-Abstract.The benchmarks are organized by MMLU subject areas.Source: Table 12 · p. 24
Loading 2505.23628v3…