Source-linked AI summary
Few-shot Knowledge Graph-to-Text Generation with Pretrained Language Models
Junyi Li, Tianyi Tang, Wayne Xin Zhao, Zhicheng Wei, Nicholas Jing Yuan, Ji-Rong Wen
TL;DR
Few-shot KG-to-text generation aims to describe graph facts when collecting large labeled datasets is difficult. The paper combines pretrained language models with alignment, relation-biased KG linearization, and multi-task learning; it reports best performance in both fully supervised and few-shot settings.
Problem
Large-scale labeled KG–text pairs are costly to collect, motivating generation from only a handful of examples.
Source: §1 · p. 1Method
Results
The model achieves the best performance in both fully supervised and few-shot settings.
Source: §5.2 · p. 7Takeaways & Limitations
Experiments on three benchmark datasets demonstrate the effectiveness of the proposed few-shot KG-to-text model.
Source: §6 · p. 9
Abstract
from arXiv · showhide
This paper studies how to automatically generate a natural language text that describes the facts in knowledge graph (KG). Considering the few-shot setting, we leverage the excellent capacities of pretrained language models (PLMs) in language understanding and generation. We make three major technical contributions, namely representation alignment for bridging the semantic gap between KG encodings and PLMs, relation-biased KG linearization for deriving better input representations, and multi-task learning for learning the correspondence between KG and text. Extensive experiments on three benchmark datasets have demonstrated the effectiveness of our model on KG-to-text generation task. In particular, our model outperforms all comparison methods on both fully-supervised and few-shot settings. Our code and datasets are available at https://github.com/RUCAIBox/Few-Shot-KG2Text.
1 Introduction
KG-to-text generation turns graph facts into descriptive text, but few-shot use is difficult because annotations are costly and PLMs face a gap between structured KG inputs and text. The paper proposes a PLM-based approach and reports experiments on three benchmark datasets.
1 Introduction
- Few-shot KG-to-text generation targets satisfactory descriptive text from only several hundred labeled instances, addressing the expense of collecting large-scale KG–text pairs.The task is motivated by the impracticality of producing large labeled datasets across domains.Source: §1 · p. 1
- PLMs may help because their pretraining encodes world knowledge and supports few-shot language understanding and generation.The paper presents this capacity as potentially beneficial for understanding and describing KG facts.Source: §1 · p. 1
- Two challenges are bridging the semantic gap between text-pretrained PLMs and structured KG inputs, and learning accurate KG–text correspondence with limited labels.Source: §1 · pp. 1–2
- The model combines KG–PLM representation alignment, relation-biased breadth-first linearization, and joint text generation and KG reconstruction.The authors state that reconstruction further enhances semantic correspondence between the input KG and output text.Source: §1 · p. 2
- The authors report this as the first study of PLMs for few-shot KG-to-text generation and report effectiveness on three benchmark datasets.Source: §1 · p. 2
2 Related Work
Prior KG-to-text work spans rule-based generation, linearized triples, and models that explicitly encode graph structure. This paper focuses on PLMs for few-shot generation while accounting for KG structure.
2 Related Work
- Early systems used grammar rules, while later neural systems linearized triples but could not model KG structural information.Source: §2 · p. 2
- Graph-aware approaches include graph Transformers over local neighborhoods and GNNs combining global and local message passing.The cited examples are Koncel-Kedziorski et al. (2019) and Ribeiro et al. (2020a).Source: §2 · p. 2
- CycleGT uses unsupervised iterative back-translation between text and graph data.Source: §2 · p. 2
- The paper distinguishes its approach by exploring large PLMs for few-shot KG-to-text generation while bridging KG encodings and PLMs.Source: §2 · p. 2
- Most PLMs are conditioned on text, and prior PLM-based KG-to-text work randomly linearized graphs into triples without explicitly modeling KG structure.The paper identifies structural relations as important for faithful text generation.Source: §2 · p. 2
3 Problem Formulation
The task takes a compact KG subgraph and generates concise, correct text representing its entities and relations. The formulation also identifies entity mentions and a masked-text input used for representation alignment.
3 Problem Formulation
- A KG is a set of triples ⟨e, r, e′⟩, where entities e and e′ are connected by relation r.The input is a small, compact subgraph extracted from a larger knowledge graph.Source: §3 · p. 2
- The target is a natural-language sequence representing the correct and concise semantics of the entities and relations in the input KG.Source: §3 · p. 2
- Entity mentions are recorded with their entity and start and end positions; only the first mention of each entity is retained.Source: §3 · p. 2
- Replacing each mention’s words with [MASK] produces masked text, which is also used as input for representation alignment.Source: §3 · p. 2
- The few-shot setting uses only a handful of training instances, such as 200, with a given PLM such as GPT-2.Source: §3 · p. 3
4 Approach
The approach addresses input representation and KG–text correspondence through representation alignment, relation-biased graph linearization, and joint generation and reconstruction objectives.
4.1 Representation Alignment
- A GNN encodes KG entities, and representation alignment minimizes distance to PLM-based entity embeddings so KG representations can enter the PLM.The KG encoder uses R-GCN to leverage multi-relational information; entities are represented at the token level to support generalization to unseen entities.Source: §4.1 · pp. 3–4
4.2 Knowledge Graph Linearization
- Relation-biased breadth-first search orders same-layer entities by relation-based weights, producing a linearized entity sequence for the decoder.The strategy reflects the observation that some relations, such as nationality, are lexicalized before others.Source: §4.2 · p. 4
4.3 KG-enhanced Multi-task Learning
- Training combines text generation and entity copying with KG reconstruction, which the authors say helps capture KG–text correspondence and generate faithful text.The reconstruction task predicts relations between entities in generated text.Source: §4.3 · pp. 4–5
4.4 Discussion and Learning
- For few-shot unseen entities, BPE splits entity words into smaller semantic units; target-text encoding and representation alignment are used only during training.The model uses BART-Large, while the target text and text encoder are not used at test time.Source: §4.4 · p. 5
5 Experiments
Experiments compare the model with KG-to-text baselines on three benchmarks in fully supervised and few-shot settings. The reported findings favor the model overall and examine its components, linearization strategies, human-rated outputs, and generated examples.
5.1 Experimental Setup
- Experiments use AGENDA, WebNLG, and GenWiki Fine, with five specified WebNLG and GenWiki domains and few-shot training sizes from 50 to 500.Comparison methods are trained on the whole training set.Source: §5.1 · pp. 5–6
5.2 Main Results
- The model achieves the best performance in fully supervised and few-shot settings, including improvements over CGE-LW and CycleGT with 500 labelled instances.Baselines drop drastically with few examples, while the model’s performance descends only slightly.Source: Table 2 · p. 6, Table 3 · p. 6, Table 4 · p. 6
5.3 Detailed Analysis
- Removing any loss lowers BLEU, ROUGE, and CIDEr, with larger drops for representation alignment and KG reconstruction than for copy loss.The authors suggest PLM-learned phrase expressions may explain the smaller drop without copy loss.Source: Table 5 · p. 7, §5.3 · p. 7
- On WEBNLG, RBFS and FFS outperform the other tested traversal strategies; searching by layer may yield more coherent entity sequences.The comparison reports BLEU results and includes RDFS and random search as alternatives.Source: Figure 3 · p. 7, §5.3 · pp. 7–8
- In human evaluation of 200 generated examples, three workers scored factual correctness and naturalness, and the authors report more faithful, fluent text than previous models.Factual correctness counts supported, contradicted, and missing facts; naturalness concerns grammatical correctness and fluency.Source: Table 6 · p. 7, §5.3 · p. 8
- Generated examples show content sketches similar to reference texts, while the authors report that their model describes KG facts correctly where BART-Large merges or misses facts.They also note entity words connected by meaningful keywords in their generated text.Source: §5.3 · pp. 8–9, Table 7 · p. 8
6 Conclusion
The conclusion presents a PLM-based few-shot KG-to-text model built around representation alignment, relation-biased linearization, and multi-task learning. Experiments on three benchmarks demonstrate its effectiveness; KG-enhanced PLMs are proposed as future work.
6 Conclusion
- The model’s three contributions are representation alignment between KG encodings and PLMs, relation-biased KG linearization, and multi-task learning for KG-text correspondence.Source: §6 · p. 9
- Experiments on three benchmark datasets demonstrate the effectiveness of the few-shot KG-to-text generation model.Source: §6 · p. 9
- Future work will consider KG-enhanced PLMs that explicitly inject knowledge information into PLMs to improve task performance.Source: §6 · p. 9