Source-linked AI summary
Neural data-to-text generation: A comparison between pipeline and end-to-end architectures
Thiago Castro Ferreira, Chris van der Lee, Emiel van Miltenburg, Emiel Krahmer
TL;DR
Data-to-text generation can use either modular neural pipelines or end-to-end models, but their relative effectiveness had not been systematically established. This paper compares both architectures on RDF-to-text generation using GRU and Transformer models, finding that explicit intermediate supervision yields better texts and somewhat better generalization to unseen domains.
Problem
The paper addresses whether neural end-to-end data-to-text models outperform neural pipelines with explicit intermediate generation steps.
Method
The study implements and systematically compares neural pipeline and end-to-end RDF-to-text architectures using GRU and Transformer models with automatic, human, and qualitative evaluation.
Results
Pipeline models produce better results than fully end-to-end approaches, offer somewhat better generalization to unseen domains, and compare favorably with the current state of the art.
Takeaways & Limitations
Explicit intermediate steps and supervision are supported as a more effective design than fully end-to-end generation for this RDF-to-text setting.
Takeaways & Limitations
The experiments use WebNLG, and adaptation of the RDF-triple pipeline to other domains and languages remains future work.
Abstract
from arXiv · showhide
Traditionally, most data-to-text applications have been designed using a modular pipeline architecture, in which non-linguistic input data is converted into natural language through several intermediate transformations. In contrast, recent neural models for data-to-text generation have been proposed as end-to-end approaches, where the non-linguistic input is rendered in natural language with much less explicit intermediate representations in-between. This study introduces a systematic comparison between neural pipeline and end-to-end data-to-text approaches for the generation of text from RDF triples. Both architectures were implemented making use of state-of-the art deep learning methods as the encoder-decoder Gated-Recurrent Units (GRU) and Transformer. Automatic and human evaluations together with a qualitative analysis suggest that having explicit intermediate steps in the generation process results in better texts than the ones generated by end-to-end approaches. Moreover, the pipeline models generalize better to unseen inputs. Data and code are publicly available.
1 Introduction
The paper compares neural pipeline and end-to-end architectures for converting RDF triples into text, addressing whether explicit intermediate supervision improves generation. Using GRU and Transformer models with automatic, human, and qualitative evaluation, it finds that pipeline models produce better results and generalize somewhat better to unseen domains.
- 1 Introduction: The comparison addresses a previously unstudied trade-off between pipeline error cascading and the potential benefits of specialized neural modules.Pipeline errors can cascade across stages, while dedicated modules may improve individual tasks and produce more reusable outputs.
- 1 Introduction: The study systematically compares neural pipeline and end-to-end generation from RDF triples using GRU and Transformer architectures.The pipeline performs discourse ordering, text structuring, lexicalization, referring expression generation, and textual realization sequentially, whereas the end-to-end model maps triples directly to text.
- 1 Introduction: Pipeline models achieve better results than fully end-to-end approaches when separate generation modules provide supervision.The comparison uses automatic and human measures together with qualitative analysis.
- 1 Introduction: The pipeline architecture offers somewhat better generalization to unseen domains and compares favorably with the current state of the art.
2 Data
The experiments use WebNLG, a semantically varied RDF triple-to-text corpus with held-out domains, augmented with gold-standard representations for intermediate pipeline tasks.
- 2 Data: WebNLG contains 25,298 texts describing 9,674 sets of up to 7 RDF triples across 15 domains, including 5 domains held out from training and validation.
- 2 Data: Figure 1 illustrates how a set of RDF triples corresponds to a target text.
- 2 Data: The augmented corpus provides gold-standard representations for discourse ordering, text structuring, lexicalization, and referring expression generation.
3 Pipeline Architecture
The pipeline converts RDF triples into text through sequential ordering, structuring, lexicalization, referring-expression generation, and textual-realization steps.
- 3 Pipeline Architecture: The proposed pipeline converts RDF triples into text through 5 sequential steps.
- 3.1 Discourse Ordering: Discourse Ordering determines the sequence in which input triples are verbalized.The model predicts an ordering over linearized triples, which is then used to retrieve the triples in that order.
- 3.2 Text Structuring: Text Structuring organizes ordered triples into paragraphs and sentences by segmenting predicates with sentence-boundary tokens.Because WebNLG contains single-paragraph texts, evaluation focuses on sentence planning.
- 3.3 Lexicalization: Lexicalization predicts templates that verbalize predicates and encode information for later reference generation and textual realization.The templates use ENTITY tags for entity references and VP and DT tags for verb and determiner features.
- 3.4 Referring Expression Generation: Referring Expression Generation realizes entity references in predicted templates, using NeuralREG with bidirectional context encoders and attention.
- 3.5 Textual Realization: Textual Realization converts the intermediate representation into grammatical text by setting verb and determiner forms with rule-based strategies.This final step is not individually evaluated.
4 End-to-End Architecture
The end-to-end architecture directly maps an unordered linearized set of RDF triples to text without explicit intermediate representations.
- 4 End-to-End Architecture: The end-to-end model directly converts an unordered linearized set of RDF triples into natural-language text.It follows neural data-to-text approaches based on Neural Machine Translation techniques and does not explicitly model intermediate representations.
5 Models Set-Up
The study models both pipeline steps and end-to-end generation with GRU and Transformer encoder-decoder architectures. GRUs process tokens sequentially, whereas Transformers encode the source sequence as a whole using positional embeddings.
- Both pipeline steps and end-to-end generation use GRU and Transformer encoder-decoder models.
- GRUs encode input tokens sequentially, while Transformers encode the source sequence as a whole with positional embeddings.
- The architectures therefore differ in recurrent structure and how they represent source order.
6 Experiment 1: Learning the pipeline
The first experiment evaluates neural models as separate pipeline modules for discourse ordering, text structuring, lexicalization, and referring expression generation. Neural models perform better on seen domains but generally lose to simpler baselines on unseen domains.
- Experiment design: The experiment evaluates deep-learning methods for Discourse Ordering, Text Structuring, Lexicalization, and Referring Expression Generation against simpler baselines.
- Data: Gold-standard intermediate representations from the augmented WebNLG corpus train and evaluate the pipeline steps.
- Evaluation: Discourse Ordering and Text Structuring are scored by accuracy, Referring Expression Generation by accuracy, and Lexicalization by BLEU.
- Results: Neural pipeline models outperform the baselines on seen domains but generally drop substantially below them on unseen domains.Text Structuring is the exception: neural models beat Majority on unseen domains but remain below Random.
- Results: GRUs outperform Transformers in Discourse Ordering and Text Structuring, whereas Transformers perform better in Lexicalization.
7 Experiment 2: Pipeline vs. End-to-End
The second experiment compares neural pipeline and end-to-end RDF-to-text systems using automatic, human, and qualitative evaluations. Across these analyses, pipeline systems generally produce stronger and more faithful texts, especially on unseen domains.
- Evaluation: The evaluation combines BLEU and METEOR, human fluency and semantic ratings, and qualitative measures of predicate retention and grammatical mistakes.
- Automatic Evaluation: Neural GRU and Transformer pipeline models outperform all reference approaches in BLEU across domains, while different systems lead on seen and unseen subsets.The end-to-end GRU and Random pipeline obtain the best results on seen and unseen domains, respectively, in the reported comparison.
- Human Evaluation: Neural pipeline systems receive the highest fluency ratings in seen domains, while UPF-FORGe leads fluency and semantic ratings in unseen domains.
- Qualitative Analysis: UPF-FORGe verbalizes the input triples in 91% of evaluated trials and contains no annotator-identified grammatical mistakes.
- Qualitative Analysis: Within neural pipelines, Transformers retain all input triples in 68% of trials versus 67% for GRUs, while fluency differs little.
- Qualitative Analysis: Less than half of end-to-end trials verbalize all input triples, and their outputs often contain more information than the non-linguistic input.
8 Discussion
The study finds that neural pipeline approaches generally outperform end-to-end models, especially on unseen domains, where end-to-end systems generalize poorly and may hallucinate content. Explicit intermediate representations therefore remain advantageous for RDF-to-text generation, although reference-system comparisons depend on whether unseen domains were anticipated.
- Individual pipeline modules exceeded Random and Majority baselines on seen domains but fell below those baselines on unseen domains.
- GRUs performed better for discourse ordering and text structuring, while Transformers performed better at verbalizing ordered and structured triples.
- Pipeline models generated more fluent, more faithful texts than end-to-end models across most tested conditions, with the largest gap on unseen domains.The authors conclude that end-to-end approaches do not generalize as well as pipeline approaches.
- On an unseen four-triple example, pipeline systems produced semantically related text from the seen predicates, whereas end-to-end systems hallucinated unrelated content.
- UPF-FORGe achieved the best results on most metrics, but its design included both seen and unseen domains, limiting the fairness of direct comparison.When evaluated only on seen domains, the authors report that their neural pipelines were rated higher on almost all metrics.
- The conclusion attributes better fluency and faithfulness than full end-to-end approaches to adding supervision through separate generation modules.
A Models Set-Up
The models used shared training and decoding settings across GRU and Transformer architectures, with architecture-specific configurations for recurrent and self-attention components. The pipeline additionally used BPE segmentation and NeuralREG for referring-expression generation.
- Both architectures used Nematus, were trained three times, and selected development-set-best settings for testing.
- Training used Adam with early stopping, tied embeddings, beam search of size 5, and output sequences capped at 100 tokens.
- GRU models used bidirectional attention-based recurrent layers with 300D embeddings, 512D hidden units, layer normalization, and dropout.
- Transformer models used six encoder and decoder layers, 512D embeddings and hidden units, eight attention heads, 2048D feed-forward layers, dropout, warm-up, and label smoothing.
- BPE was applied to pipeline lexicalization targets and end-to-end texts using 20,000 learned merge operations.
- The pipeline generated referring expressions with a concatenative-attention NeuralREG model and used rule-based handling for verbs and determiners.