PaperlayerView exact revision on arXiv

Source-linked AI summary

Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction

Bowen Zhang, Harold Soh

arXiv:2404.03868v2cs.CLcs.AIcs.LG

TL;DR

Existing LLM-based KGC methods are difficult to scale to large schemas and cannot assume a predefined schema is available. EDC addresses these issues with open extraction, schema definition, and post-hoc canonicalization, optionally aided by schema retrieval. Across three KGC benchmarks, it extracts high-quality triplets with larger schemas than prior methods and supports both predefined and automatically generated schemas.

  • Problem

    LLM-based KGC methods struggle with schemas too large for prompts, while some applications lack a fixed predefined schema.

  • Method

    EDC combines open information extraction, schema definition, and post-hoc canonicalization, with a trained Schema Retriever for relevant schema elements.

  • Results

    EDC extracts higher-quality knowledge graphs than state-of-the-art methods across target-alignment and self-canonicalization settings, while Schema Retriever use significantly and consistently improves performance.

  • Takeaways & Limitations

    EDC supports high-quality knowledge graph construction with large schemas or without a predefined schema.

  • Takeaways & Limitations

    EDC does not address entity deduplication and is costly because it involves many LLM calls.

Abstract

from arXiv · show

In this work, we are interested in automated methods for knowledge graph creation (KGC) from input text. Progress on large language models (LLMs) has prompted a series of recent works applying them to KGC, e.g., via zero/few-shot prompting. Despite successes on small domain-specific datasets, these models face difficulties scaling up to text common in many real-world applications. A principal issue is that, in prior methods, the KG schema has to be included in the LLM prompt to generate valid triplets; larger and more complex schemas easily exceed the LLMs' context window length. Furthermore, there are scenarios where a fixed pre-defined schema is not available and we would like the method to construct a high-quality KG with a succinct self-generated schema. To address these problems, we propose a three-phase framework named Extract-Define-Canonicalize (EDC): open information extraction followed by schema definition and post-hoc canonicalization. EDC is flexible in that it can be applied to settings where a pre-defined target schema is available and when it is not; in the latter case, it constructs a schema automatically and applies self-canonicalization. To further improve performance, we introduce a trained component that retrieves schema elements relevant to the input text; this improves the LLMs' extraction performance in a retrieval-augmented generation-like manner. We demonstrate on three KGC benchmarks that EDC is able to extract high-quality triplets without any parameter tuning and with significantly larger schemas compared to prior works. Code for EDC is available at https://github.com/clear-nus/edc.

1 Introduction

Knowledge graphs organize interconnected information as nodes and edges.

  • Knowledge graphs represent interconnected information through graph structures, with entities and relations as nodes and edges.

EDC: Extract-Define-Canonicalize

EDC addresses scalable knowledge graph construction by separating open extraction from schema definition and canonicalization. It supports both large predefined schemas and settings where the schema must be generated automatically, with retrieval improving extraction quality.

  • Existing LLM-based KGC methods struggle with large schemas because schema information must fit in the prompt, and fixed schemas are not always available.
  • EDC decomposes knowledge graph construction into open information extraction, schema definition, and schema canonicalization.The framework first extracts triplets without a fixed schema, defines schema components, and standardizes the resulting triplets.
  • EDC supports predefined target schemas and automatically generates a schema when none is provided.
  • EDC extracts higher-quality knowledge graphs than state-of-the-art methods across three datasets and both target-alignment and self-canonicalization settings.
  • Schema Retriever retrieves schema components relevant to the input text, significantly and consistently improving EDC’s performance.

2 Background

Prior KGC methods evolved from modular pipelines toward end-to-end generation, while open information extraction and canonicalization address schema flexibility and triplet standardization.

  • Traditional KGC pipelines separately handled entity discovery, entity typing, and relation classification, whereas newer methods generate triplets end to end.
  • Open information extraction avoids the requirement for a predefined schema, but canonicalization is needed to standardize triplets for downstream applications.
  • Canonicalization differs by setting: target-schema alignment maps to existing components, while schema-free methods consolidate components without a target schema.
  • EDC works with or without a target schema and uses LLM-generated contextual definitions with verification to reduce over-generalization.

3 Method: EDC for KGC

EDC structures knowledge graph construction as extraction, schema definition, and canonicalization, with optional refinement using retrieved schema components. Its canonicalization strategy changes depending on whether a target schema exists.

  • EDC first extracts schema-independent relational triplets, then defines their schema components and canonicalizes them into a less ambiguous, less redundant form.
  • With a target schema, EDC aligns components to likely matches and excludes transformations that the LLM judges semantically unreasonable.
  • Without a target schema, EDC consolidates similar components while adding non-transformable components to an expanding canonical schema.
  • In the worked example, canonicalization transforms relations into birthDate and mission, producing a canonicalized knowledge graph.
  • The refinement phase supplies prior entities and relations plus retrieved schema relations as extraction hints, bootstrapping the next extraction round.
  • Schema Retriever projects text and schema components into a relevance space where cosine similarity identifies schema components likely to occur in the input.

4 Experiments

EDC and EDC+R are evaluated across three KGC datasets under Target Alignment, with comparisons spanning multiple LLMs, baselines, and schema settings. EDC generally matches or exceeds baselines, refinement improves performance, and annotation incompleteness affects evaluation on REBEL and Wiki-NRE.

  • Results and Analysis: EDC significantly outperforms state-of-the-art methods in both Target Alignment and Self Canonicalization settings, while refinement further improves EDC.The experiments use three KGC datasets and report automatic and manual evaluations.
  • Experimental Setup: The experiments use WebNLG, REBEL, and Wiki-NRE because their relation-type diversity better reflects real-world complexity than several smaller-schema alternatives.The datasets induce schemas with 159, 200, and 45 unique relation types, respectively.
  • Target Alignment: EDC is superior or on par with state-of-the-art baselines across WebNLG, REBEL, and Wiki-NRE in Partial F1 comparisons.Figure 2 compares EDC and EDC+R with different LLMs for open information extraction against the respective baselines.
  • Target Alignment: EDC’s performance is significantly better on WebNLG than on REBEL and Wiki-NRE, where incomplete or extraneous reference triplets can penalize valid extractions.REBEL and Wiki-NRE use distant supervision, which can produce incomplete reference sets and overly pessimistic evaluations.
  • Target Alignment: The schema retriever improves EDC by identifying relevant relations that LLMs may miss during open information extraction.The ablation removes retrieved relations during refinement and reports a performance decline.

5 Conclusion

EDC combines open information extraction with post-hoc canonicalization to construct knowledge graphs. Experiments show that EDC and EDC+R outperform specialized trained models with a target schema and can dynamically create one without a provided schema.

  • EDC is an LLM-based three-phase framework that addresses knowledge graph construction through open information extraction followed by post-hoc canonicalization.
  • EDC and EDC+R extract better knowledge graphs than specialized trained models with a target schema and dynamically create a schema when none is provided.

6 Limitations and Future Directions

The authors identify limitations involving entity duplication, component quality, model coverage, computational cost, and future applications. They propose entity de-duplication, broader retriever training, smaller-model evaluation, and component replacement as future directions.

  • EDC does not include entity de-duplication, so constructed knowledge graphs may retain redundant entities.The authors suggest coreference resolution as a future mechanism.
  • EDC’s components can be improved, including the schema retriever through training on more diverse and higher-quality data.
  • The experiments varied LLMs only for open information extraction, while the other EDC modules relied on GPT-3.5-turbo.The authors propose testing smaller open-source models on the remaining tasks.
  • EDC cost around 0.009 USD per example when GPT-3.5-turbo was used for all components.The authors suggest replacing some components with smaller fine-tuned models.
  • The authors plan to apply EDC to embodied AI and robotics, where knowledge graphs could serve as memory sources for vision-language models.

7 Ethical Considerations

The paper reports dataset licensing and annotator-protection practices, while recognizing risks from current LLMs including hallucinations and privacy issues. The experiments use GPT-3.5-turbo, GPT-4, and Mistral-7b with few-shot prompting.

  • The datasets were used for research under their corresponding licenses, and the code is publicly available.The authors note that the datasets may contain information about individuals, especially celebrities.
  • Two university-student annotators were compensated fairly, given flexible time, and covered by an institutionally exempt collection protocol.
  • The authors identify hallucinations and privacy issues as potential risks of using current LLMs.
  • The study uses GPT-3.5-turbo, GPT-4, and the 7-billion-parameter open-source model Mistral-7b.
  • EDC modules use few-shot prompting with six examples, while canonicalization and refinement retrieve top-5 and top-10 schema relations, respectively.These hyperparameters were chosen to balance performance and inference costs.

A.3 Schema Retriever Training

The schema retriever is trained to distinguish relevant relations from negatives using text-relation pairs and an InfoNCE objective. Its refinement hints combine extracted and retrieved entities and relations to improve triplet extraction.

  • The schema retriever is fine-tuned on text-relation pairs to distinguish the correct relation from non-relevant relations using InfoNCE loss.The training data contains 37,500 text-relation pairs derived from TEKGEN, evenly divided between positive and negative samples.
  • The training objective uses cosine similarity between a text and candidate relations, with negative samples represented by N.
  • Recall@10 reached 0.823 on WebNLG, 0.663 on REBEL, and 0.818 on Wiki-NRE.
  • Refinement combines entities from the previous EDC iteration with entities separately extracted from the input text.The merged candidates are supplied as hints for refined triplet extraction.
  • Candidate relations combine prior extracted relations with schema-retrieved relations and their definitions.Definitions help the LLM assess retrieved relations when the retriever returns irrelevant information.
  • The refinement hint recovered the fine-grained selectedByNasa relation and helped avoid noisy relations through semantically rich descriptions.

C.1 Complete Results

Across WebNLG, REBEL, and Wiki-NRE, EDC matches or exceeds state-of-the-art baselines across evaluation criteria, while refinement further improves results. Additional analyses examine iterative refinement, extraction coverage, and annotation limitations.

  • Complete benchmark results: EDC performs better than or comparably to state-of-the-art baselines across Precision, Recall, and F1 under Partial, Strict, and Exact criteria, while EDC+R consistently improves further.The complete results cover WebNLG, REBEL, and Wiki-NRE; EDC+R uses one refinement iteration in the main results.
  • Iterative refinement: Further refinement improves results stably but yields diminishing returns, motivating the use of only one refinement iteration in the main results.Table 6 reports results for an additional refinement iteration.
  • Refinement ablation: Removing entities and relations from the previous round’s refinement hint worsens coverage, supporting iterative merging of information across refinement rounds.The ablation attributes better KGC to improved coverage of entities and relations in the text.
  • Annotation analysis: EDC extracts significantly more triplets than reference annotations and GENIE on REBEL and Wiki-NRE, with manual evaluation finding many additional triplets meaningful and correct.Incomplete annotations can make automatic scores overly pessimistic, although EDC still exceeds the baseline by a large margin.
  • Novel dataset: EDC and EDC+R remain superior to GenIE on a novel fictional dataset created to reduce the risk of training-data contamination.The fictional dataset contains 50 annotated entries using the Wiki-NRE schema.

E Comparison against previous LLM-based approaches

On smaller-schema datasets, EDC alone can lag behind prior LLM-based methods, but refinement substantially improves results. Relation definitions help resolve ambiguous relations and head-tail errors, while EDC already performs strongly on the moderately sized Wiki-NRE schema.

  • Comparison with prior LLM methods: On CONLL and SciERC, EDC alone may not surpass baselines, whereas EDC+R achieves significantly better results.The comparison uses GPT-3.5-turbo and includes CONLL, SciERC, and Wiki-NRE.
  • Refinement effects: Relation definitions help disambiguate homonyms such as “follows” and correct head-tail relation errors such as identifying which argument is the father.These error-correcting effects are attributed to the refinement step and were not possible in previous methods.
  • Wiki-NRE comparison: On Wiki-NRE, EDC significantly outperforms baseline methods, while ChatIE and CodeKGC may still produce out-of-schema relations despite receiving the full schema.The passage links this result to possible confusion from long context.

F Combine EDC with other IE tools

EDC can be combined with chunking and coreference resolution for long-document extraction, substantially improving strict micro F1 over direct prompting. Entity deduplication provides a smaller additional gain, while combining extraction and schema definition was explored to reduce cost and latency.

  • Chunking and coreference: Strict micro F1 rises from 0.132 to 0.234 when EDC is combined with LingMess coreference resolution and sentence-level chunking, compared with 0.060 for direct prompting.The combination targets long documents that exceed LLM context windows.
  • Entity deduplication: Entity deduplication with CESI slightly improves REBEL Partial F1 from 0.516 to 0.520.CESI is applied as a post-hoc canonicalization method to EDC’s resulting KGs.
  • Cost reduction: Combining OIE and Schema Definition was explored to reduce latency and cost because separate LLM calls increase both.The original separation allowed a more expensive model for OIE and a cheaper model for schema definition.
Loading 2404.03868v2…