PaperlayerView exact revision on arXiv

Source-linked AI summary

Investigating Pretrained Language Models for Graph-to-Text Generation

Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Schütze, Iryna Gurevych

arXiv:2007.08426v3cs.CL

TL;DR

Graph-to-text generation must produce fluent descriptions from graph data, while the effectiveness of pretrained language models and task-adaptive pretraining remains under study. The paper evaluates BART and T5 across three graph domains using adaptive pretraining strategies, finding state-of-the-art results and evidence that memorized facts can support generation even with reduced graph structure.

  • Problem

    The paper investigates how pretrained language models and task-adaptive pretraining affect graph-to-text generation across multiple graph domains.

  • Method

    The study compares BART and T5 with language model adaptation or supervised task adaptation before fine-tuning on graph-to-text benchmarks.

  • Results

    BART and T5 outperform prior state-of-the-art systems across three benchmarks, with task-adaptive pretraining providing further gains.

  • Takeaways & Limitations

    PLMs can perform well on KG-to-text tasks even with shuffled bags of node and edge labels, suggesting memorized facts guide generation.

  • Takeaways & Limitations

    Qualitative analysis finds that PLMs may parrot training sentences while ignoring input structure, limiting practical use when faithfulness matters.

Abstract

from arXiv · show

Graph-to-text generation aims to generate fluent texts from graph-based data. In this paper, we investigate two recently proposed pretrained language models (PLMs) and analyze the impact of different task-adaptive pretraining strategies for PLMs in graph-to-text generation. We present a study across three graph domains: meaning representations, Wikipedia knowledge graphs (KGs) and scientific KGs. We show that the PLMs BART and T5 achieve new state-of-the-art results and that task-adaptive pretraining strategies improve their performance even further. In particular, we report new state-of-the-art BLEU scores of 49.72 on LDC2017T10, 59.70 on WebNLG, and 25.66 on AGENDA datasets - a relative improvement of 31.8%, 4.5%, and 42.4%, respectively. In an extensive analysis, we identify possible reasons for the PLMs' success on graph-to-text tasks. We find evidence that their knowledge about true facts helps them perform well even when the input graph representation is reduced to a simple bag of node and edge labels.

1 Introduction

Graph-to-text generation converts graph-based meaning or factual structures into fluent text for applications including dialogue and question answering. This paper studies BART and T5 across three graph domains, adding task-adaptive pretraining and examining how much models use graph structure.

  • Motivation: Graph-to-text generation aims to create fluent natural-language descriptions from input graphs.The task supports dialogue generation and question answering, where graphs can represent dialog states or database-query knowledge.
  • Approach: The study evaluates encoder-decoder PLMs BART and T5 across meaning representations, Wikipedia KGs, and scientific KGs.These architectures are selected for conditional text generation.
  • Results: Task-adaptive pretraining improves state-of-the-art graph-to-text performance across three established benchmarks.The investigated strategies include language model adaptation and supervised task adaptation using additional task-specific data.
  • Analysis: PLMs maintain high performance on two KG-to-text benchmarks when graphs are reduced to shuffled bags of node and edge labels.This representation removes explicit connectivity information, prompting analysis of how models exploit graph linearizations and memorized knowledge.
  • Results: 2.6 to 12.0 BLEU points separate the approaches from prior state-of-the-art systems across three graph-to-text benchmarks.The approaches also exceed specialized graph architectures such as graph neural networks.

2 Related Work

Prior graph-to-text work includes linearized graph inputs, sequence-based KG-to-text models, graph neural networks, and Transformer architectures with injected structure. This paper distinguishes itself by studying BART and T5 with task-adaptive pretraining and analyzing why PLMs perform well.

  • Graph-to-text Learning: Early AMR-to-text and KG-to-text systems generated text from linearized graphs or sequences of knowledge-graph triples.These approaches treated graph information as an input sequence rather than necessarily encoding graph structure directly.
  • Graph-to-text Learning: Later systems used GNNs or injected graph-structure information into Transformer-based architectures.This line of work explicitly targets structural encoding for graph-to-text generation.
  • Pretrained Language Models: BART and T5 were introduced as encoder-decoder PLMs for AMR-to-text generation in contrast to earlier decoder-only GPT-2 work.Related studies also reported PLM results on other data-to-text datasets.
  • Task Adaptation: The paper focuses on general transfer strategies using task-adaptive pretraining with additional task-specific data across PLMs and benchmarks.It also provides an analysis of PLM performance on KG-to-text tasks.
  • Task Adaptation: LMA uses domain-specific self-supervision on target texts, whereas DAPT operates on model inputs; the paper reports benefits from additional task-specific pretraining for graph-to-text.This distinction separates the paper’s adaptation setup from related text-classification work.

3 PLMs for Graph-to-Text Generation

The paper fine-tunes BART and T5 for graph-to-text generation and inserts an intermediate task-adaptive pretraining stage using relevant text or graph-text pairs. It tests language model adaptation and supervised task adaptation before downstream fine-tuning.

  • PLM Setup: BART and T5 are Transformer encoder-decoder PLMs evaluated for conditional graph-to-text generation.Experiments vary model capacity, including small, base, and large configurations.
  • PLM Setup: The standard training procedure fine-tunes both PLMs for several epochs on supervised graph-to-text datasets.T5 receives a task prefix before the graph input.
  • Task-specific Adaptation: Task-specific adaptation adds an intermediate pretraining step between original pretraining and graph-to-text fine-tuning.The additional corpus is more task-relevant and usually smaller than general pretraining corpora.
  • Task-specific Adaptation: LMA continues language-model pretraining on reference texts, while STA uses graph-text pairs from the same or a similar domain.LMA excludes graphs and uses only the task-specific target texts to adapt the decoder.

4 Datasets

The study evaluates graph-to-text generation on AMR, WebNLG, and AGENDA, spanning meaning representations, Wikipedia knowledge graphs, and scientific knowledge graphs. Task-specific data are collected for AMR-like and scientific domains, but not WebNLG.

  • AMR: AMR represents sentence meaning as a rooted directed graph whose nodes are concepts and edges are semantic relations.LDC2017T10 pairs each annotated sentence with its corresponding AMR graph, linearized using PENMAN notation.
  • WebNLG: WebNLG pairs DBPedia knowledge graphs with one or more descriptive sentences and separates test instances into seen, unseen, and all partitions.The model input marks heads, relations, and tails with special tokens.
  • AGENDA: AGENDA pairs automatically constructed scientific KGs with paper titles and abstracts, using entities as nodes and relations as edges.Its loose graph-text alignments make the dataset more challenging for graph-to-text generation.
  • Task-specific Data: Task-specific data are collected for AMR-like meaning representations and scientific data, but not WebNLG because its texts were specifically written by annotators.AMR silver graphs come from parsed Gigaword sentences, while scientific data come from Semantic Scholar papers and extracted KGs.

5 Experiments

Across AMR, WebNLG, and AGENDA, BART and T5 achieve strong graph-to-text performance, with task-adaptive pretraining providing further gains where evaluated. Human evaluation and controlled analyses indicate that these models generate fluent text and can perform well with limited or obscured graph structure.

  • AMR-to-text: 49.72 BLEU makes T5 the new state of the art on AMR-to-text after supervised task adaptation.Task-adaptive pretraining also accelerates convergence; T5large with STA converges after 2 epochs instead of 5 without additional pretraining.
  • WebNLG: 59.70 BLEU makes T5 the new state of the art on WebNLG, including 65.05 on seen and 54.69 on unseen test partitions.End-to-end models generally perform worse on unseen data, whereas models with explicit graph structure perform best among non-PLM approaches.
  • AGENDA: 25.66 BLEU makes task-adapted BART the new AGENDA state of the art, gaining 2 BLEU points over its non-adapted performance.The authors hypothesize that loose graph-text coverage makes AGENDA more challenging and limits adaptation gains relative to AMR.
  • Human evaluation: Human evaluation finds statistically significant improvements in fluency, meaning similarity, and semantic adequacy over other state-of-the-art approaches on AMR and WebNLG.The largest fluency improvement is +0.97 on AMR, where the BLEU improvement over Harkous et al. is +8.10; generated texts also score above references in fluency.
  • Data efficiency: With 40% of training data, both BART and T5 greatly improve performance relative to full-data training across all three benchmarks, while T5 is more data-efficient at 1%.For example, BART reaches 91% of its full-data AMR BLEU with 40% of the training data; at 1%, T5 leads BART by 7.51 AMR and 5.64 WebNLG BLEU points.

6 Influence of the Graph Structure

The experiments test how much PLMs use graph structure by comparing ordered and shuffled inputs. KG-to-text performance remains relatively robust without structure, suggesting memorized facts and language modeling contribute strongly, while ignoring structure risks unfaithful generation.

  • Experimental setup: Shuffling removes graph structure by randomizing sequences of node and edge labels, turning the input into a bag of labels.The comparison uses T5order for correctly ordered triples and T5shuf for shuffled triples during both training and evaluation.
  • Results by graph domain: AMR-to-text performance drops significantly when graph structure is removed.With neutral separators, correctly ordered inputs lose only around 2–4% relative to representations using explicit tags or parentheses.
  • Results by graph domain: KG-to-text performance is not much lower for shuffled inputs, indicating that PLM success relies more on language modeling than graph encoding.The authors hypothesize that models match entities in shuffled inputs with sentences and facts seen during pretraining or fine-tuning.
  • Qualitative analysis: T5shuf can generate text with the same content as T5order, showing that correct triple structure is not always necessary.Even T5order produces reasonable and truthful text, but the models can rely more on language modeling than graph structure.
  • Qualitative analysis: Controlled corruptions show that T5order transfers a false two-triple fact reliably, whereas T5shuf does not.This supports the observation that the correctly ordered model relies somewhat more on input graph structure.
  • Practical limitation: PLMs risk parroting training sentences while ignoring input structure, which can limit practical use when outputs must remain faithful to the graph.The limitation is identified through qualitative analysis of fluent generations.

7 Conclusion

The paper concludes that task-adaptive pretraining improves PLM-based graph-to-text generation across three benchmarks, while memorized facts can guide generation when graph structure is not followed.

  • Conclusion: Language model adaptation and supervised task adaptation produce notable improvements for BART and T5.The approaches outperform the state of the art by a substantial margin on three graph-to-text benchmarks.
  • Conclusion: PLMs may use memorized facts instead of consistently following graph structure during generation.The paper proposes injecting a stronger graph-structural bias to preserve language-modeling strengths while improving faithfulness.
  • Conclusion: Human evaluators perceive the generated texts as significantly more fluent than human references.This finding is reported alongside the benchmark improvements.

Appendices

The supplementary material adds experimental-data information and reports results omitted from the main paper.

  • Appendices: The supplementary material provides additional information about the experimental data.
  • Appendices: It also includes results that were not included in the paper’s main body.

A AMR Input Representation

The appendix compares three AMR linearizations and selects PENMAN for subsequent experiments because it performs best on the development set.

  • AMR representations: The study evaluates node-only, depth-first-search, and PENMAN representations for the input AMR graph.Node-only removes edge information, while DFS and PENMAN linearize the graph with edge relations.
  • AMR representations: The node-only representation retains only concept values and removes edge information.
  • AMR representations: The DFS representation traverses the graph while retaining labeled semantic relations.
  • AMR representations: The PENMAN representation expresses nested concepts and relations using parentheses and relation labels.
  • AMR representation selection: PENMAN achieves the best results on the AMR development set with T5small and is therefore used in later experiments.

B Cross-domain Adaptation

Cross-domain task-adaptive pretraining can improve graph-to-text performance even when related data are unavailable or come from different domains. BART benefits in most cross-domain cases, while T5 pretrained on KGAIA improves WebNLG performance.

  • BARTbase gains performance in most cross-domain settings with task-adaptive pretraining.
  • T5base pretrained on KGAIA improves performance on WebNLG.
  • Cross-domain task-adaptive pretraining improves graph-to-text performance even when the additional data come from a different domain.The paper motivates this setting because closely related data are not always available.

C Input Graph Size

Input graph size changes how much graph structure is reflected in T5small’s WebNLG performance. Ordered and shuffled inputs behave similarly for one-triple graphs, but diverge as graphs become larger.

  • C Input Graph Size: T5order and T5shuf perform similarly on WebNLG inputs containing one triple.The comparison concerns T5small evaluated on seen and unseen WebNLG test settings.
  • C Input Graph Size: The performance gap between T5order and T5shuf increases with larger WebNLG graphs.The results suggest that graph structure is taken into account more for graphs with more than 2 triples.
  • C Input Graph Size: AMR graphs have more nodes and edges and substantially larger diameters and shortest paths than WebNLG and AGENDA graphs.The graph statistics use Levi-transformed undirected graphs; disconnected WebNLG and AGENDA graphs use their largest subgraph for diameter calculation.
Loading 2007.08426v3…