Source-linked AI summary
Text Generation from Knowledge Graphs with Graph Transformers
Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, Hannaneh Hajishirzi
TL;DR
The paper addresses coherent multi-sentence generation from automatically extracted knowledge graphs, avoiding the expense of manually produced document plans. It introduces GraphWriter, a graph-transformer encoder-decoder for scientific text, and reports more informative, better-structured outputs than competing methods, while noting coverage and repetition limitations.
Problem
Coherent multi-sentence generation needs structured content representations, but manually producing them is prohibitively expensive and extracted graphs introduce structural challenges.
Method
GraphWriter encodes a title and automatically extracted knowledge graph with a bidirectional title encoder and novel Graph Transformer, then generates text using attention and entity copying.
Results
Graph-structured knowledge improves generation, and GraphWriter outperforms GAT across metrics while receiving better human judgments for document structure and grammar than EntityWriter.
Takeaways & Limitations
Automatically extracted knowledge can support multi-sentence text generation, with graph structuring improving generation performance over alternative encoder-decoder setups.
Takeaways & Limitations
40% of graph entities are absent from generated text, and 18% of generated sentences repeat sentences or clauses requiring pruning.
Abstract
from arXiv · showhide
Generating texts which express complex ideas spanning multiple sentences requires a structured representation of their content (document plan), but these representations are prohibitively expensive to manually produce. In this work, we address the problem of generating coherent multi-sentence texts from the output of an information extraction system, and in particular a knowledge graph. Graphical knowledge representations are ubiquitous in computing, but pose a significant challenge for text generation techniques due to their non-hierarchical nature, collapsing of long-distance dependencies, and structural variety. We introduce a novel graph transforming encoder which can leverage the relational structure of such knowledge graphs without imposing linearization or hierarchical constraints. Incorporated into an encoder-decoder setup, we provide an end-to-end trainable system for graph-to-text generation that we apply to the domain of scientific text. Automatic and human evaluations show that our technique produces more informative texts which exhibit better document structure than competitive encoder-decoder methods.
1 Introduction
Generating coherent, multi-sentence scientific text remains difficult because document structure is complex and manually structured inputs are expensive. This work uses automatically extracted knowledge graphs and introduces GraphWriter, a graph encoder-decoder evaluated against competing systems.
- Multi-sentence generation requires overall coherence, discourse-relatedness, and carefully ordered explanations, especially in scientific writing.
- Structured inputs can guide longer-text generation, but manually annotated tables are expensive and limited to particular domains.
- Information extraction systems provide broad-domain, cross-sentence relational annotations but introduce errors, structural variety, and abstraction from surface syntax.
- GraphWriter represents extracted knowledge as a graph and extends Transformer encoding with graph structure and global information.
- The evaluations show that graph-structured knowledge improves generation over other encoder-decoder setups, while GraphWriter outperforms graph attention networks.
2 Related Work
Earlier data-to-text systems commonly generated from manually structured tables and smaller datasets, while graph-to-text research explored increasingly sophisticated graph encoders. This work targets multi-sentence scientific generation from knowledge graphs whose structure differs from denser, rooted semantic graphs.
- Earlier concept-to-text systems generated summaries from tables or records, often using manually assembled datasets and structured generation models.
- Wikipedia infobox work generated first sentences, whereas this paper addresses multi-sentence generation with relations extending beyond sentence boundaries.
- Neural models have struggled against templates on basketball statistics, motivating graph inputs that capture local and global structure.
- AMR systems often linearize graphs or use encoders designed for denser, rooted, connected graphs, unlike the knowledge graphs studied here.
- The paper differs from title-conditioned abstract generation by using extracted information as input and proposing a model intended across tasks and domains.
3 The AGENDA Dataset
The paper frames abstract generation as producing title-appropriate text that expresses an automatically extracted knowledge graph. It introduces AGENDA, pairs scientific abstracts with SciIE-derived graphs, and standardizes its data splits.
- Knowledge graphs represent global and local information through entity neighborhoods and paths connecting distant nodes through intermediate entities.
- AGENDA pairs scientific article titles and abstracts with knowledge graphs to evaluate title-appropriate, graph-content-preserving abstract generation.
- SciIE extracts scientific entity types, coreference annotations, and seven relations for each abstract.
- Coreferential entities are collapsed into longest-mention nodes, and relation annotations become labeled edges in possibly unconnected graphs.
- The standardized split contains 38,720 training, 1,000 validation, and 1,000 test datapoints.
4 Model
GraphWriter encodes a restructured knowledge graph with a Graph Transformer and generates text through an attention-based encoder-decoder with copying. The model combines local graph attention, global context, title encoding, and end-to-end training.
- GraphWriter architecture: GraphWriter encodes a title with a bidirectional recurrent network and a knowledge graph with a novel Graph Transformer.The decoder attends to both encodings during generation.
- Graph preparation: Disconnected labeled knowledge graphs are converted into connected unlabeled graphs by replacing each labeled edge with forward and reverse relation vertices.A global vertex connects entity vertices to promote information flow between disconnected graph components.
- Graph Transformer: The Graph Transformer contextualizes each vertex by multi-headed self-attention over its graph neighborhood and adds mechanisms for capturing global graph context.Stacked blocks propagate information through the graph, producing graph-contextualized vertex encodings.
- Graph Transformer: Each Transformer block combines neighborhood attention, residual connections, layer normalization, and a two-layer feedforward network.The blocks are stacked repeatedly so information can propagate through the graph.
- Decoder: At every decoding step, attention produces graph and title context vectors that are concatenated and supplied to an input-feeding decoder.The decoder can either generate a vocabulary token or copy an entity from the knowledge graph or title.
5 Experiments
Experiments compare GraphWriter with graph, entity-only, title-only, and alternative graph encoders using automatic and expert human evaluations. GraphWriter generally performs best, while analyses show benefits from global graph contextualization and structured knowledge, alongside coverage and repetition errors.
- Evaluation setup: The evaluation combines automatic metrics with expert Best-Worst Scaling judgments of generated scientific abstracts.Fifteen computer science students provided expert judgments; BLEU and METEOR supplemented human evaluation.
- Automatic evaluation: GraphWriter outperforms the comparison systems on automatic metrics, while models using titles, entities, and relations outperform models using less information.The baselines include GAT, EntityWriter, and Rewriter.
- Automatic evaluation: GraphWriter outperforms GAT across metrics in four training runs, with non-overlapping variances between the models.The authors interpret this result as evidence that GraphWriter’s global contextualization improves generation.
- Human evaluation: Expert judgments select GraphWriter as best more often and worst less often than Rewriter, supporting the value of including knowledge in generation.Some generated texts were preferred to human abstracts, partly because canonicalized entity mentions introduce disfluencies in the reference texts.
- Human evaluation: Graph structure improves generation over an unstructured entity collection, with the largest human-judgment gains in document structure and grammar.The result suggests that input structure is reflected in the generated surface form.
- Output analysis: GraphWriter uses more input entities in articulated contexts and repeats less than GAT, while EntityWriter and Rewriter show weaker coherence or input relatedness.Rewriter remains fluent and grammatical but jumps between topics; 40% of graph entities are omitted and 18% of generated sentences require repetition pruning.
6 Conclusion
The paper shows that incorporating knowledge as graphs improves multi-sentence generation from automatic information extraction systems. It introduces GraphWriter, provides the AGENDA dataset, and identifies repetition and entity coverage as future challenges.
- GraphWriter incorporates knowledge as graphs and improves multi-sentence text generation from automatic information extraction systems.
- The work evaluates GraphWriter against strong baselines using human and automatic evaluations.
- The paper introduces AGENDA, a dataset of scientific abstracts paired with extracted knowledge.
- Future work could address repetition and entity coverage in generated texts.