PaperlayerView exact revision on arXiv

Source-linked AI summary

Schema-adaptable Knowledge Graph Construction

Hongbin Ye, Honghao Gui, Xin Xu, Xi Chen, Huajun Chen, Ningyu Zhang

arXiv:2305.08703v4cs.CLcs.AIcs.DBcs.IRcs.LG

TL;DR

Fixed-schema information extraction is difficult to apply when schemas evolve, motivating schema-adaptable KGC and a benchmark spanning three expansion patterns. The paper proposes ADAKGC, whose prefix instructor and dynamic decoding condition extraction on the schema; experiments demonstrate effectiveness, while the authors note limited generalization remains.

  • Problem

    Existing extraction systems use fixed, predefined schemas, while evolving schemas require models to adapt without retraining.

    Source: §1 · p. 1

  • Method

    The paper defines schema-adaptable KGC, builds horizontal, vertical, and hybrid expansion benchmarks, and proposes ADAKGC with schema-enriched prefix instructions and schema-conditioned dynamic decoding.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    ADAKGC demonstrates effectiveness on the three schema-expansion patterns, although performance tends to decrease as iterations increase.

    Source: §4.2 · p. 6, §6 · p. 9

  • Takeaways & Limitations

    The benchmark and baseline provide a setting for studying knowledge extraction under evolving schemas, which the authors describe as an underexplored task.

    Source: §6 · p. 9

    ADAKGC still suffers from poor generalization ability despite outperforming previous approaches.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Conventional Knowledge Graph Construction (KGC) approaches typically follow the static information extraction paradigm with a closed set of pre-defined schema. As a result, such approaches fall short when applied to dynamic scenarios or domains, whereas a new type of knowledge emerges. This necessitates a system that can handle evolving schema automatically to extract information for KGC. To address this need, we propose a new task called schema-adaptable KGC, which aims to continually extract entity, relation, and event based on a dynamically changing schema graph without re-training. We first split and convert existing datasets based on three principles to build a benchmark, i.e., horizontal schema expansion, vertical schema expansion, and hybrid schema expansion; then investigate the schema-adaptable performance of several well-known approaches such as Text2Event, TANL, UIE and GPT-3.5. We further propose a simple yet effective baseline dubbed \textsc{AdaKGC}, which contains schema-enriched prefix instructor and schema-conditioned dynamic decoding to better handle evolving schema. Comprehensive experimental results illustrate that AdaKGC can outperform baselines but still have room for improvement. We hope the proposed work can deliver benefits to the community. Code and datasets available at https://github.com/zjunlp/AdaKGC.

1 Introduction

Conventional extraction systems depend on fixed schemas, while real-world schemas evolve, making existing models difficult to use without retraining. The paper introduces schema-adaptable KGC and ADAKGC to address this setting.

1 Introduction

  • Fixed-schema extraction models are difficult to reuse as schemas evolve, while prior incremental-learning work emphasizes avoiding forgetting rather than ontology-evolution generalizability.The paper identifies dynamic generalizability as a key requirement for schema-evolving scenarios.Source: §1 · pp. 1–2
  • The paper defines schema-adaptable KGC and builds benchmarks for horizontal, vertical, and hybrid schema expansion across NER, RE, and EE.Text2Event, TANL, UIE, and GPT-3.5 do not effectively extract information under complex evolving schemas.Source: §1 · p. 2
  • ADAKGC combines schema-enriched prefix instructors for schema-specific knowledge with schema-conditioned dynamic decoding that rebuilds a trie when schemas change.The approach is described as model agnostic.Source: §1 · p. 2
  • The paper contributes a new schema-adaptable KGC task and benchmark, plus the ADAKGC baseline and experiments on its adaptability.The authors describe the task as underexplored and the benchmark as presenting new challenges and research opportunities.Source: §1 · p. 2

2 Problem Statement and Overview

The paper frames KGC as extracting structured knowledge from text under a schema that may change over time. It formalizes evaluation without retraining, defines three schema-expansion patterns, and describes the benchmark setting and baselines.

2.1 Background of KGC

  • KGC extracts knowledge from unstructured text under a defined schema; the paper focuses on predicting structured knowledge instances from text.Prior benchmarks commonly use well-defined schemas for domain-specific knowledge graphs or structured-data integration.Source: §2.1 · p. 2

2.2 Definition of Schema-adaptable KGC

  • Schema-adaptable KGC evaluates whether a model trained on an initial schema can produce schema-constrained instances for successive schemas without retraining.The setup provides evaluation data and a schema graph at each iteration.Source: §2.2 · p. 2

2.3 Dataset Construction Process

  • The benchmark models horizontal expansion with new same-level classes, vertical expansion with subclasses, and hybrid expansion with a mixture of both.These patterns probe transfer to neighboring classes, inheritance and derivation, and potential co-evolutionary patterns, respectively.Source: §2.3 · p. 3

2.4 Schema-adaptable KGC Benchmark

  • The benchmark highlights dynamic schema injection and schema-adjusted outputs as challenges, and considers TANL, Text2Event, UIE, and GPT-3 as baselines.TANL implicitly extracts schema information, Text2Event injects it through constrained decoding, and UIE uses schema-based prompting.Source: §2.4 · pp. 3–4

3 The Proposed Baseline: ADAKGC

ADAKGC combines a pretrained encoder-decoder with learned task and schema prefixes and schema-constrained decoding. Its prefix instructions adapt to evolving schemas, while decoding uses the current schema to constrain generated types.

3.1 Overview

  • A pretrained T5 encoder-decoder takes the schema and input together and is updated with a log-likelihood objective at each iteration.The model conditions each generated token on the preceding tokens, schema, and input.Source: §3.1 · p. 4

3.2 Schema-enriched Prefix Instructor

  • The prefix instructor uses continuous, differentiable task prompts rather than discrete natural-language instructions to guide the language model.The paper says discrete instructions may lead to suboptimal generated sequences.Source: §3.2 · p. 4
  • Schema-specific prefixes linearize relation-schema triples into dynamically adjustable constraints, combined with task prefixes for the encoder and decoder.Training first tunes task-specific prompts, then schema-specific constraints, and finally jointly optimizes these with the language model.Source: §3.2 · pp. 4–5

3.3 Schema-conditioned Dynamic Decoding

  • The decoder uses a trie built from the latest schema to constrain generated type tokens to valid schema types.The paper motivates this because greedy decoding can produce outdated or invalid types when the schema changes.Source: §3.3 · p. 5

4 Experiments

Across the three schema-expansion settings, performance tends to decline over iterations, while ADAKGC achieves the best performance in most settings. Ablations and case studies show both the contribution and remaining challenges of schema adaptation.

4.1 Experimental Settings

  • Experiments cover NER, RE, and EE on FEW-NERD, NYT, and ACE2005, using span-based Micro-F1 and averages over three random seeds.UIE uses T5-v1.1-base without pre-training for the comparison.Source: §4.1 · p. 5

4.2 Main Results

  • Across horizontal, vertical, and hybrid expansion, performance tends to decrease as iterations increase; ADAKGC and UIE can perform optimally or near-optimally but drop from iteration 1 to 7.TANL has lower performance, while Text2Event outperforms other models in some iterations.Source: Table 1 · p. 6, Table 2 · p. 6, §4.2 · p. 6

Schema-enhanced modules boost the perfor-

  • GPT-3.5 can produce instances conforming to changing schemas in 20-shot prompting, but its performance remains low due to the low-shot setting.The authors also report stable generalization in sampled cases evaluated with ChatGPT3.Source: Figure 4 · p. 6, Schema-enhanced modules boost the… · p. 7
  • ADAKGC achieves the best performance in most settings with schema-enhanced encoder and decoder modules; it improves 0.71% on trigger extraction and 3.65% on event argument extraction.The reported improvements are on the ACE2005 hybrid schema expansion dataset.Source: Schema-enhanced modules boost the… · p. 6, Table 3 · p. 7

4.3 Ablation Study on ADAKGC

  • Both schema-enriched prefix instructors and schema-conditioned dynamic decoding help; efficiently encoding schema evolution information is more important, with improvements of 0.77% horizontally and 0.36% vertically.The ablation averages results over seven iterations on the NERD horizontal schema expansion dataset.Source: Figure 5 · p. 7, §4.3 · p. 7

4.4 Case Study

  • In vertical expansion, initial-schema types mostly degrade in performance as the schema grows, while inherited parent labels can transfer to newly added child types.The case study compares iterations 1 and 7 and randomly selects eight types.Source: Figure 6 · p. 7, §4.4 · p. 7
  • The error analysis groups mistakes into weak transfer, inheritance deficiency, relevance neglect, class imbalance, and potential annotation errors or omissions.The categories are reported across ACE2005 schema expansion datasets.Source: Table 4 · p. 8, §4.4 · p. 8

5 Related Work

Prior KGC work includes schema-constrained extraction, open extraction, and low-resource approaches. Lifelong learning addresses continual class learning, while this paper frames schema expansion through three principles.

5.1 Knowledge Graph Construction

  • Open knowledge extraction avoids schema constraints, whereas schema-driven KGC uses evolutionary instructions and ontological associations to address schema adaptation.The authors distinguish schema-driven adaptation from instance-driven KGC in low-resource settings.Source: §5.1 · p. 8

5.2 Lifelong Learning

  • Lifelong-learning methods address continual class learning through regularization, replay, architecture, or distillation; prior work focused on class increments, while this paper discusses three schema-expansion principles.The cited work includes incremental methods for NER, relation extraction, and event detection.Source: §5.2 · p. 8

6 Conclusion

The paper introduces schema-adaptable KGC, benchmark datasets, and ADAKGC, demonstrating task difficulty for prior baselines and effectiveness of its baseline across three schema-expansion patterns.

  • The paper introduces schema-adaptable KGC with benchmark datasets and ADAKGC, illustrating task difficulty for prior baselines and ADAKGC's effectiveness across horizontal, vertical, and hybrid schema expansion.Source: §6 · p. 9

Limitations

The benchmark dataset coverage and ADAKGC generalization remain limited. The authors identify restricted dataset availability and simple schema patterns as constraints, and note that ADAKGC still generalizes poorly.

Datasets

  • Previous datasets may have simple schema patterns, and ACE2005 cannot be released because of licensing restrictions.The authors plan to build more datasets through crowdsourcing.Source: Limitations · p. 9
  • The authors plan to promote multimodal schema-adaptive graphs integrating visual and textual knowledge.They describe this as future work toward a self-learning graph extraction system.Source: Limitations · p. 9

Baselines and Proposed ADAKGC

  • ADAKGC still suffers from poor generalization ability despite outperforming Text2Event, TANL, and UIE.The authors also describe LLM performance as stable but deficient.Source: Limitations · p. 9

Ethical Considerations

The dataset and model are intended for exploratory analysis of schema-adaptable KGC, and collected data may contain offensive or discriminatory language.

Intended use

Biases

  • Data collected from existing datasets may include offensive language or discriminatory content.Few-NERD is cited as an example source and carries a CC BY-SA 4.0 license.Source: Ethical Considerations · p. 9

A Appendix

The appendix describes experiment details, including dataset construction and evaluation on downstream tasks.

  • The appendix covers experimental details, including dataset construction and downstream-task evaluation.Source: A Appendix · p. 13

A.1.1 Construction Process

The appendix details schema-expansion dataset construction, evaluation, and implementation, then reports analogous-expansion and chatbot experiments. Its results include ADAKGC gains over Text2Event on event extraction and chatbot adaptation to a horizontally expanded schema.

A.1.1 Construction Process

  • Horizontal expansion adds same-level nodes selected through ontology similarity; vertical expansion adds child nodes, while hybrid expansion combines both strategies.Analogous expansion replaces schema nodes with semantically similar candidates selected using NPMI consistency.Source: §A.1.1 · p. 14

A.2 Evaluation

  • Span-based Micro-F1 is the primary metric, with task-specific correctness requiring matching entity, relation, trigger, or argument strings and types.Source: §A.2 · p. 14

A.1.2 Schema-adaptable Datasets Statistic

  • Table 6 reports initial subclasses, subclasses added per iteration, and total iterations for the schema-adaptable datasets.Source: Table 6 · p. 15

A.3 Hyper-parameters

  • Experiments use T5-v1.1-base, and the learning-rate search space is [1e-4, 3e-4, 5e-4] for each dataset.The implementation uses batch size 16 on one NVIDIA 3090 GPU.Source: §A.3 · p. 15

A.4 Analogous Schema Expansion Experiment

  • ADAKGC improves over Text2Event by 7.70% on event-trigger extraction and 4.87% on event-argument extraction in analogous schema expansion.Source: §A.4 · p. 15

A.5 GPT-3.5 Experiment Details

  • The GPT-3.5 experiment uses a fixed prompt template with task descriptions, linearized schemas, 20 demonstrations, and task prompts.Source: §A.5 · p. 15

A.6 ChatGPT Results

  • ChatGPT extracts some facts under the demonstrated schema and adapts its output after three new nodes are added in a horizontal expansion iteration.The authors report that it also deduces reasonable facts through a chain-of-thought approach.Source: §A.6 · p. 16, Figure 8 · p. 24

Related papers

Loading 2305.08703v4…