PaperlayerView exact revision on arXiv

Source-linked AI summary

Text-to-Text Pre-Training for Data-to-Text Tasks

Mihir Kale, Abhinav Rastogi

arXiv:2005.10433v3cs.CL

TL;DR

Data-to-text generation requires producing natural language from structured sources, and the paper studies whether large-scale text-to-text transfer learning improves this task. It fine-tunes T5 on linearized structured data and finds state-of-the-art performance, stronger out-of-domain robustness, and advantages over pipelined systems and BERT or GPT-2 alternatives. The authors present this approach as a baseline while identifying tailored unsupervised objectives and multilingual extension as future directions.

  • Problem

    The paper examines whether large-scale text-to-text transfer learning can improve generation from structured data such as tables, graphs, and meaning representations.

    Source: §1 · p. 1

  • Method

    The approach linearizes structured data into flat strings and fine-tunes pre-trained T5 encoder-decoder models on data-to-text corpora.

    Source: §4 · p. 2

  • Results

    T5 fine-tuning achieves state-of-the-art results, improves robustness to out-of-domain inputs, and outperforms pipelined approaches and BERT or GPT-2 alternatives.

    Source: §1 · p. 1, §7 · p. 5

  • Takeaways & Limitations

    A simple end-to-end T5 model provides a useful baseline for data-to-text research as pre-training becomes more prevalent.

    Source: §1 · p. 1, §7 · p. 5

    The approach is simple and leaves tailored unsupervised objectives, unlabelled-data use, and multilingual extension for future work.

    Source: §1 · p. 1, §7 · p. 5

Abstract

from arXiv · show

We study the pre-train + fine-tune strategy for data-to-text tasks. Our experiments indicate that text-to-text pre-training in the form of T5, enables simple, end-to-end transformer based models to outperform pipelined neural architectures tailored for data-to-text generation, as well as alternative language model based pre-training techniques such as BERT and GPT-2. Importantly, T5 pre-training leads to better generalization, as evidenced by large improvements on out-of-domain test sets. We hope our work serves as a useful baseline for future research, as transfer learning becomes ever more prevalent for data-to-text tasks.

1 Introduction

The paper studies large-scale text-to-text transfer learning for data-to-text generation, where structured sources such as tables and graphs are verbalized. Fine-tuned T5 models achieve strong benchmark performance, improve out-of-domain robustness, and outperform several alternative approaches.

  • Data-to-text generation produces natural-language text conditioned on structured sources such as tables or graphs.Applications include task-oriented dialogue and weather-forecast summarization.Source: §1 · p. 1
  • T5 fine-tuning achieves state-of-the-art results across MultiWoz, ToTTo, and WebNLG.These benchmarks cover task-oriented dialogue, tables-to-text, and graph-to-text generation.Source: §1 · p. 1
  • Pre-training greatly improves robustness to out-of-domain inputs.Source: §1 · p. 1
  • A simple end-to-end transformer can outperform sophisticated multi-stage pipelines and graph neural network architectures when pre-trained.Source: §1 · p. 1
  • T5 outperforms alternative language-model pre-training approaches such as BERT and GPT-2.Source: §1 · p. 1
  • The approach is presented as a useful baseline, while the authors identify tailored unsupervised objectives and broader use of unlabelled data as future directions.Source: §1 · p. 1

2 Related Work

Prior data-to-text research progressed from rule-based systems toward neural and pipelined approaches. The paper situates its contribution against methods that condition language models on tables, model entities explicitly, or encode structured data with graph convolutions.

  • Early data-to-text systems were primarily rule-based, whereas recent work has increasingly favored neural approaches.Source: §2 · p. 1
  • Recent methods condition language models on tables, explicitly model entities, or represent structured data with graph convolutional networks.Source: §2 · p. 1
  • Some prior studies reported that neural pipelined approaches outperform end-to-end models.Source: §2 · p. 1
  • The datasets span different structured-data formats, including graphs or triples, as illustrated alongside their linearized representations and target texts.Source: Figure 1 · p. 2

3 Pre-training

The paper uses released T5 encoder-decoder models pre-trained on unsupervised and supervised text-to-text tasks. It evaluates model capacity across four T5 sizes for data-to-text fine-tuning.

  • T5 uses a transformer encoder-decoder architecture pre-trained with unsupervised span masking on Common Crawl data.Source: §3 · p. 2
  • Its multitask pre-training includes supervised translation, summarization, classification, and question answering, but not structured-data generation.Source: §3 · p. 2
  • The experiments compare T5-Small, Base, Large, and 3B models containing 60 million, 220 million, 770 million, and 3 billion parameters, respectively.Source: §3 · p. 2

4 Fine-tuning

The data-to-text task is converted into T5’s text-to-text format by linearizing structured inputs and fine-tuning the model on each corpus. Training uses standardized optimization and greedy decoding, with a preliminary unsupervised comparison showing no performance difference.

  • Structured data is represented as a flat linearized string, then T5 is fine-tuned on the data-to-text corpus.Source: §4 · p. 2
  • Fine-tuning uses a constant learning rate of 0.001, a batch size of 131,072 tokens, and a maximum input length of 512 tokens.Source: §4 · p. 2
  • WebNLG training runs for 5K steps, while ToTTo training runs for 10K steps.Source: §4 · p. 2
  • Initial experiments with purely unsupervised T5 variants showed no performance difference.Source: §4 · p. 2
  • All model parameters are updated during fine-tuning, and the best checkpoint is selected by development-set BLEU.Source: §4 · p. 2
  • Decoding uses greedy search, and final evaluation applies the metrics used by prior work for each dataset.Source: §4 · p. 2

5 Datasets

Experiments cover three English data-to-text datasets spanning multiple domains and structured-data formats. They include tables, task-oriented dialogue meaning representations, and graphs or triples.

  • The experiments span three English datasets across a variety of domains.Source: §5 · p. 2
  • ToTTo pairs selected Wikipedia table cells and metadata, such as the table title, with natural-language descriptions.Source: §5 · p. 2
  • MultiWoz is a 10K human-human dialogue corpus whose NLG task verbalizes system-action meaning representations into natural-language responses.Source: §5 · p. 2
  • WebNLG converts graphs of subject-object-predicate triples into textual descriptions.Source: §5 · p. 2
  • The datasets use different structured-data inputs: tables, meaning representations, and graphs or triples.Source: §5 · p. 2

6 Results and Discussion

Across WebNLG, ToTTo, and MultiWoz, T5-based models achieve strong results and generalize better to out-of-domain inputs. Results also show that model capacity and evaluation setting affect performance.

WebNLG

  • T5-Large improves over DualEnc by 4.3 BLEU overall and 14 BLEU on WebNLG’s unseen test set.T5-Large performs best across BLEU and METEOR, with stronger gains on new domains and relations.Source: §6.1 · p. 3

Overall Findings

  • T5-based end-to-end neural models surpass sophisticated pipelined approaches while being more robust to domain shift.The compared systems include planning, graph-based, discourse-ordering, lexicalization, and referring-expression-generation pipelines.Source: §6.1 · p. 3

ToTTo

  • T5-3B2 improves over the BERT baseline by 5.5 BLEU and 5.8 PARENT on ToTTo.On the Non-Overlap out-of-domain test set, the improvements increase to 6.6 BLEU and 7.5 PARENT.Source: §6.2 · p. 4

MultiWoz

  • All T5 models, including T5-Small with 5x fewer parameters, outperform SC-GPT2 by 4-5 BLEU on MultiWoz.T5 achieves this without in-domain pre-training, although its SER is slightly worse than SC-GPT2’s.Source: §6.3 · p. 4

Human Evaluation

  • On WebNLG’s unseen test set, DualEnc drops 24% in accuracy while T5 scores 90% accuracy and 2.33 fluency.Human evaluation uses three raters per prediction and reports majority-vote accuracy and average naturalness.Source: §6.4 · p. 4

Impact of Model Capacity

  • WebNLG unseen-test performance rises with model size, including a 10 BLEU jump from T5-Small to T5-Base.The results suggest that capacity is critical for out-of-domain generalization; ToTTo shows a similar pattern with smaller gains beyond T5-Base.Source: §6.5 · pp. 4–5

7 Conclusion

T5 pre-training achieves state-of-the-art data-to-text results while greatly improving robustness to out-of-domain inputs. The authors identify task-specific unsupervised objectives and multilingual extension, especially for low-resource languages, as future directions.

  • T5 pre-training leads to state-of-the-art results for the data-to-text task.Source: §7 · p. 5
  • T5 greatly improves robustness to out-of-domain inputs.Source: §7 · p. 5
  • Future work includes unsupervised pre-training objectives tailored to data-to-text and extensions to multiple languages, especially low-resource ones.Source: §7 · p. 5

Related papers

Loading 2005.10433v3…