Source-linked AI summary
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke Zettlemoyer
TL;DR
Existing pretraining methods often target particular end tasks, limiting their applicability across tasks. BART addresses this with denoising sequence-to-sequence pretraining and achieves similar discriminative-task performance to RoBERTa alongside state-of-the-art results on multiple generation tasks.
Problem
Existing pretraining methods typically target particular end tasks, limiting their applicability across a wider range of tasks.
Method
BART pretrains a sequence-to-sequence denoising autoencoder to reconstruct original documents from corrupted text using a bidirectional encoder and autoregressive decoder.
Results
BART achieves similar performance to RoBERTa on discriminative tasks and new state-of-the-art results on multiple text-generation tasks, including a 6 ROUGE improvement on XSum.
Takeaways & Limitations
BART provides a broadly applicable pretraining approach that is particularly effective for text generation while also performing well on comprehension tasks.
Takeaways & Limitations
BART is less effective when outputs are only loosely constrained by the input, as on ELI5, where a pure language model performs best.
Abstract
from arXiv · showhide
We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and many other more recent pretraining schemes. We evaluate a number of noising approaches, finding the best performance by both randomly shuffling the order of the original sentences and using a novel in-filling scheme, where spans of text are replaced with a single mask token. BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks. It matches the performance of RoBERTa with comparable training resources on GLUE and SQuAD, achieves new state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains of up to 6 ROUGE. BART also provides a 1.1 BLEU increase over a back-translation system for machine translation, with only target language pretraining. We also report ablation experiments that replicate other pretraining schemes within the BART framework, to better measure which factors most influence end-task performance.
1 Introduction
BART is a flexible sequence-to-sequence denoising autoencoder that addresses the limited applicability of task-specific pretraining methods. It performs strongly across generation, comprehension, and machine translation tasks while supporting controlled comparisons with other training objectives.
- Motivation and analysis: BART targets the limited applicability of masked-language-model variants, which typically specialize in particular end tasks such as span prediction or generation.Its ablation analysis replicates other training objectives while controlling data and optimization factors, finding consistently strong performance across the considered tasks.
- Model and training: BART pretrains a sequence-to-sequence denoising autoencoder by corrupting text with an arbitrary noising function and reconstructing the original text.Its standard Transformer-based neural machine translation architecture combines bidirectional encoding with autoregressive decoding and applies across a wide range of end tasks.
- Noise functions: The best noising performance combines random sentence-order shuffling with span infilling, replacing arbitrary-length spans, including zero-length spans, with one mask token.This flexible corruption scheme generalizes original word masking and next-sentence prediction approaches.
- Empirical results: BART matches RoBERTa on GLUE and SQuAD and achieves state-of-the-art results on abstractive dialogue, question answering, and summarization, including a 6 ROUGE gain on XSum.These results use comparable training resources and show particular effectiveness for text generation alongside strong comprehension performance.
- Machine translation: BART improves a strong back-translation baseline by 1.1 BLEU on WMT Romanian-English using a target-side pretrained language model.The method stacks BART above additional Transformer layers that translate the foreign language into noised English for propagation through BART.
2 Model
BART is a sequence-to-sequence denoising autoencoder that reconstructs original documents from corrupted inputs using a bidirectional encoder and left-to-right decoder. Its flexible corruption framework supports arbitrary document transformations, trained with reconstruction cross-entropy.
- Model: BART reconstructs original documents from corrupted inputs with a bidirectional encoder, left-to-right autoregressive decoder, and negative-log-likelihood training objective.The reconstruction loss is the cross-entropy between decoder outputs and the original document.
- Architecture: The architecture follows the standard sequence-to-sequence Transformer, using GeLUs, N(0, 0.02) initialization, and 6 encoder/decoder layers in base versus 12 in large.The Transformer architecture is adapted from Vaswani et al. (2017), with GeLUs following GPT and parameter initialization from N(0, 0.02).
- Architecture: Compared with BERT, BART adds decoder cross-attention, omits BERT’s pre-word-prediction feed-forward network, and has roughly 10% more parameters at equivalent size.Each decoder layer cross-attends to the encoder’s final hidden layer.
- Model: BART applies arbitrary document corruptions before reconstruction, including previously proposed and novel transformations that can be composed.If all source information is removed, BART becomes equivalent to a language model.
- Noising transformations: Text infilling replaces sampled spans with single [MASK] tokens, with span lengths drawn from a Poisson distribution (λ = 3), while other tested corruptions include token masking, deletion, sentence permutation, and document rotation.Token masking replaces random tokens; deletion removes them; sentence permutation shuffles sentences; rotation starts the document at a randomly selected token.
3 Fine-tuning BART
BART supports downstream sequence, token, and generation tasks through task-specific uses of its encoder–decoder representations. For machine translation, the full pretrained model can serve as an English decoder by adding and training a new source encoder.
- Sequence classification: BART uses the final decoder token’s hidden state for sequence classification, allowing it to attend to decoder states from the complete input.The input is fed to both encoder and decoder, followed by a new multi-class linear classifier.
- Token classification: For token classification, BART uses decoder top hidden states as word representations to classify tokens such as SQuAD answer endpoints.The complete document is fed into both the encoder and decoder.
- Sequence generation: BART’s autoregressive decoder directly supports sequence generation tasks such as abstractive question answering and summarization.The encoder receives the input sequence, while the decoder generates outputs autoregressively; these tasks manipulate and copy input information, matching denoising pretraining.
- Machine translation: For translation into English, BART’s encoder and decoder operate as a pretrained decoder, with a newly initialized source encoder learned from bitext.The source encoder maps foreign words into inputs that BART can denoise into English and may use a separate vocabulary.
- Machine translation: The source encoder is trained in two stages: first with most BART parameters frozen, then with all model parameters updated briefly.Both stages backpropagate cross-entropy loss from BART’s output.
4 Comparing Pre-training Objectives
BART compares multiple pre-training objectives under largely controlled data, model-size, training, and fine-tuning conditions. Results show that objective effectiveness depends strongly on the downstream task, while text-infilling BART provides the most consistently strong performance except on ELI5.
- Overall results: Pre-training effectiveness varies significantly by task: a simple language model performs best on ELI5 but worst on SQuAD, whereas BART text-infilling models are strong across tasks except ELI5.The comparison uses comparable-size models trained for 1M steps, with bottom-block systems sharing data, code, and fine-tuning procedures.
- Objective trends: Token masking or deletion is central to successful objectives, while document rotation and sentence permutation perform poorly in isolation; deletion outperforms masking on generation tasks.Masked and permuted language models also underperform on generation, and neither includes left-to-right autoregressive language modeling during pre-training.
- Task-specific findings: Bidirectional encoders are crucial for SQuAD: a purely left-to-right decoder performs poorly, whereas BART achieves similar performance with half as many bidirectional layers.Future context is important for classification decisions in this task.
- Limitations: The permuted language model underperforms XLNet (Yang et al., 2019), partly because it omits relative-position embeddings and segment-level recurrence.This indicates that the pre-training objective is not the only factor affecting performance.
- Task-specific findings: Pure language models perform best on ELI5, suggesting BART is less effective when outputs are only loosely constrained by their inputs.ELI5 is an outlier with much higher perplexities than the other tasks and is the only generation task where other models outperform BART.
5 Large-scale Pre-training Experiments
Large-scale BART matches RoBERTa on discriminative tasks while achieving strong generation results, including improvements over prior work in summarization, dialogue, and abstractive question answering. The model uses RoBERTa-scale pre-training with text infilling and sentence permutation.
- Pre-training setup: BART is trained at RoBERTa scale to test large-scale performance, using the same 160Gb pre-training corpus and disabling dropout during the final 10% of training steps.The corpus contains news, books, stories, and web text; the dropout change was intended to help fit the CNN/DailyMail task.
- Pre-training setup: The large BART model has 12 encoder and decoder layers, hidden size 1024, batch size 8000, and 500000 training steps.It uses GPT-2 byte-pair encoding, masks 30% of tokens, permutes all sentences, and combines text infilling with sentence permutation.
- Summarization: BART outperforms previous work on CNN/DailyMail and XSum across both summarization datasets and all reported metrics.On XSum, BART exceeds the best prior BERT-based system by roughly 6.0 points on all ROUGE metrics; on CNN/DailyMail, it surpasses existing work despite strong extractive baselines.
- Discriminative tasks: BART performs similarly to RoBERTa on SQuAD and GLUE, with only small task-level differences, showing generation gains do not sacrifice classification performance.The comparison uses the same pre-training resources but a different objective; Table 2 also compares BART with XLNet.
- Dialogue and abstractive QA: BART outperforms previous work on CONVAI2 response generation and exceeds the best previous ELI5 system by 1.2 ROUGE-L.CONVAI2 evaluates responses conditioned on dialogue context and a textually specified persona, while ELI5 tests long freeform answers and remains challenging because questions weakly specify answers.
6 Qualitative Analysis
BART improves summarization metrics by up to 6 points over the prior state of the art, while qualitative examples show fluent, highly abstractive, generally factually accurate summaries that combine document evidence with background knowledge. These samples indicate that BART learns a strong combination of language understanding and generation.
- 6 Qualitative Analysis: BART improves summarization metrics by up to 6 points over the prior state of the art.The qualitative analysis complements these automated results by examining generated summaries.
- 6 Qualitative Analysis: BART generates fluent, grammatical English that is highly abstractive, with few phrases copied from the input.Table 7 presents examples from post-pretraining-corpus WikiNews articles after removing each article’s first sentence to prevent an easy extractive summary.
- 6 Qualitative Analysis: The outputs are generally factually accurate and integrate supporting evidence from across the document with background knowledge.Examples include correctly completing names and inferring that PG&E operates in California.
- 6 Qualitative Analysis: The samples demonstrate that BART pretraining learns a strong combination of natural language understanding and generation.
7 Related Work
Prior pretraining methods differ in context modeling, prediction structure, and suitability for generation or discrimination. BART addresses these distinctions through autoregressive decoding over bidirectional context, contrasting with BERT, UniLM, MASS, XL-Net, and multilingual translation pretraining approaches.
- Language-model pretraining: Earlier GPT and ELMo (Peters et al., 2018) use separate directional representations, while very large language models enable unsupervised multitask behavior.GPT models only leftward context; ELMo concatenates left-only and right-only representations without pretraining interactions between them.
- Language-model pretraining: BERT (Devlin et al., 2019) learns left-right contextual interactions through masked language modelling, but non-autoregressive predictions reduce its effectiveness for generation.Longer training (Liu et al., 2019), parameter sharing (Lan et al., 2019), and span masking (Joshi et al., 2019) improve BERT.
- Comparison with related sequence-to-sequence models: BART supports generative and discriminative tasks like UniLM (Dong et al., 2019), while its autoregressive decoder and uncorrupted decoder context better match generation.UniLM predictions are conditionally independent, whereas BART trains its decoder left-to-right on uncorrupted context.
- Pretraining for machine translation: Machine-translation pretraining has gained most from jointly pretraining source and target languages, though this requires pretraining on every language of interest.Other work improves encoders with pretrained representations, while decoder gains are more limited (Edunov et al., 2019).
8 Conclusions
BART pretrains by mapping corrupted documents to their originals, matching RoBERTa on discriminative tasks and achieving state-of-the-art results on several text-generation tasks.
- 8 Conclusions: BART learns to map corrupted documents to their original forms during pre-training.The authors identify future work in developing corruption methods tailored to specific end tasks.
- 8 Conclusions: BART achieves similar performance to RoBERTa on discriminative tasks.
- 8 Conclusions: BART achieves new state-of-the-art results on several text-generation tasks.
- 8 Conclusions: Future work should explore document-corruption methods tailored to specific end tasks.