Source-linked AI summary
Text-to-Text Pre-Training for Data-to-Text Tasks
Mihir Kale, Abhinav Rastogi
TL;DR
Data-to-text generation requires producing natural language from structured sources, and the paper studies whether large-scale text-to-text transfer learning improves this task. It fine-tunes T5 on linearized structured data and finds state-of-the-art performance, stronger out-of-domain robustness, and advantages over pipelined systems and BERT or GPT-2 alternatives. The authors present this approach as a baseline while identifying tailored unsupervised objectives and multilingual extension as future directions.
Problem
The paper examines whether large-scale text-to-text transfer learning can improve generation from structured data such as tables, graphs, and meaning representations.
Method
The approach linearizes structured data into flat strings and fine-tunes pre-trained T5 encoder-decoder models on data-to-text corpora.
Results
T5 fine-tuning achieves state-of-the-art results, improves robustness to out-of-domain inputs, and outperforms pipelined approaches and BERT or GPT-2 alternatives.
Takeaways & Limitations
A simple end-to-end T5 model provides a useful baseline for data-to-text research as pre-training becomes more prevalent.
Takeaways & Limitations
The approach is simple and leaves tailored unsupervised objectives, unlabelled-data use, and multilingual extension for future work.
Abstract
from arXiv · showhide
We study the pre-train + fine-tune strategy for data-to-text tasks. Our experiments indicate that text-to-text pre-training in the form of T5, enables simple, end-to-end transformer based models to outperform pipelined neural architectures tailored for data-to-text generation, as well as alternative language model based pre-training techniques such as BERT and GPT-2. Importantly, T5 pre-training leads to better generalization, as evidenced by large improvements on out-of-domain test sets. We hope our work serves as a useful baseline for future research, as transfer learning becomes ever more prevalent for data-to-text tasks.
1 Introduction
The paper studies large-scale text-to-text transfer learning for data-to-text generation, where structured sources such as tables and graphs are verbalized. Fine-tuned T5 models achieve strong benchmark performance, improve out-of-domain robustness, and outperform several alternative approaches.
- Data-to-text generation produces natural-language text conditioned on structured sources such as tables or graphs.Applications include task-oriented dialogue and weather-forecast summarization.
- T5 fine-tuning achieves state-of-the-art results across MultiWoz, ToTTo, and WebNLG.These benchmarks cover task-oriented dialogue, tables-to-text, and graph-to-text generation.
- Pre-training greatly improves robustness to out-of-domain inputs.
- A simple end-to-end transformer can outperform sophisticated multi-stage pipelines and graph neural network architectures when pre-trained.
- T5 outperforms alternative language-model pre-training approaches such as BERT and GPT-2.
- The approach is presented as a useful baseline, while the authors identify tailored unsupervised objectives and broader use of unlabelled data as future directions.
2 Related Work
Prior data-to-text research progressed from rule-based systems toward neural and pipelined approaches. The paper situates its contribution against methods that condition language models on tables, model entities explicitly, or encode structured data with graph convolutions.
- Early data-to-text systems were primarily rule-based, whereas recent work has increasingly favored neural approaches.
- Recent methods condition language models on tables, explicitly model entities, or represent structured data with graph convolutional networks.
- Some prior studies reported that neural pipelined approaches outperform end-to-end models.
- The datasets span different structured-data formats, including graphs or triples, as illustrated alongside their linearized representations and target texts.
3 Pre-training
The paper uses released T5 encoder-decoder models pre-trained on unsupervised and supervised text-to-text tasks. It evaluates model capacity across four T5 sizes for data-to-text fine-tuning.
- T5 uses a transformer encoder-decoder architecture pre-trained with unsupervised span masking on Common Crawl data.
- Its multitask pre-training includes supervised translation, summarization, classification, and question answering, but not structured-data generation.
- The experiments compare T5-Small, Base, Large, and 3B models containing 60 million, 220 million, 770 million, and 3 billion parameters, respectively.
4 Fine-tuning
The data-to-text task is converted into T5’s text-to-text format by linearizing structured inputs and fine-tuning the model on each corpus. Training uses standardized optimization and greedy decoding, with a preliminary unsupervised comparison showing no performance difference.
- Structured data is represented as a flat linearized string, then T5 is fine-tuned on the data-to-text corpus.
- Fine-tuning uses a constant learning rate of 0.001, a batch size of 131,072 tokens, and a maximum input length of 512 tokens.
- WebNLG training runs for 5K steps, while ToTTo training runs for 10K steps.
- Initial experiments with purely unsupervised T5 variants showed no performance difference.
- All model parameters are updated during fine-tuning, and the best checkpoint is selected by development-set BLEU.
- Decoding uses greedy search, and final evaluation applies the metrics used by prior work for each dataset.
5 Datasets
Experiments cover three English data-to-text datasets spanning multiple domains and structured-data formats. They include tables, task-oriented dialogue meaning representations, and graphs or triples.
- The experiments span three English datasets across a variety of domains.
- ToTTo pairs selected Wikipedia table cells and metadata, such as the table title, with natural-language descriptions.
- MultiWoz is a 10K human-human dialogue corpus whose NLG task verbalizes system-action meaning representations into natural-language responses.
- WebNLG converts graphs of subject-object-predicate triples into textual descriptions.
- The datasets use different structured-data inputs: tables, meaning representations, and graphs or triples.
6 Results and Discussion
Across WebNLG, ToTTo, and MultiWoz, T5-based models achieve strong results and generalize better to out-of-domain inputs. Results also show that model capacity and evaluation setting affect performance.
- WebNLG: T5-Large improves over DualEnc by 4.3 BLEU overall and 14 BLEU on WebNLG’s unseen test set.T5-Large performs best across BLEU and METEOR, with stronger gains on new domains and relations.
- Overall Findings: T5-based end-to-end neural models surpass sophisticated pipelined approaches while being more robust to domain shift.The compared systems include planning, graph-based, discourse-ordering, lexicalization, and referring-expression-generation pipelines.
- ToTTo: T5-3B2 improves over the BERT baseline by 5.5 BLEU and 5.8 PARENT on ToTTo.On the Non-Overlap out-of-domain test set, the improvements increase to 6.6 BLEU and 7.5 PARENT.
- MultiWoz: All T5 models, including T5-Small with 5x fewer parameters, outperform SC-GPT2 by 4-5 BLEU on MultiWoz.T5 achieves this without in-domain pre-training, although its SER is slightly worse than SC-GPT2’s.
- Human Evaluation: On WebNLG’s unseen test set, DualEnc drops 24% in accuracy while T5 scores 90% accuracy and 2.33 fluency.Human evaluation uses three raters per prediction and reports majority-vote accuracy and average naturalness.
- Impact of Model Capacity: WebNLG unseen-test performance rises with model size, including a 10 BLEU jump from T5-Small to T5-Base.The results suggest that capacity is critical for out-of-domain generalization; ToTTo shows a similar pattern with smaller gains beyond T5-Base.
7 Conclusion
T5 pre-training achieves state-of-the-art data-to-text results while greatly improving robustness to out-of-domain inputs. The authors identify task-specific unsupervised objectives and multilingual extension, especially for low-resource languages, as future directions.
- T5 pre-training leads to state-of-the-art results for the data-to-text task.
- T5 greatly improves robustness to out-of-domain inputs.
- Future work includes unsupervised pre-training objectives tailored to data-to-text and extensions to multiple languages, especially low-resource ones.