BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

1 Introduction

• BART combines a bidirectional encoder and left-to-right decoder to reconstruct corrupted text, addressing the limited applicability of pretraining methods specialized for particular end tasks.Its standard Transformer-based translation architecture generalizes BERT through its encoder and GPT through its decoder.Source: §1 · p. 1
• Sentence shuffling combined with text infilling gives the best performance among the evaluated noising approaches.Infilling replaces arbitrary-length spans, including zero-length spans, with one mask token, allowing length-changing corruption. This generalizes BERT’s word-masking and next-sentence objectives, requiring reasoning about sentence length and longer-range input transformations.Source: §1 · p. 1
• 6 ROUGE improvement over previous work on XSum (Narayan et al., 2018) exemplifies BART’s strong generation performance.BART achieves state-of-the-art results on abstractive dialogue, question answering, and summarization. With comparable training resources, it matches RoBERTa (Liu et al., 2019) on GLUE (Wang et al., 2018) and SQuAD (Rajpurkar et al., 2016).Source: §1 · p. 1
• 1.1 BLEU improvement over a strong back-translation baseline on WMT Romanian-English demonstrates BART’s use as a pretrained target-side language model.Additional transformer layers beneath BART learn to translate foreign-language input into noised English by propagating through BART.Source: §1 · pp. 1–2
• Controlled ablations replicating other pretraining objectives find BART has the most consistently strong performance across the evaluated tasks.The comparisons control data and optimization parameters, factors reported to be as important as training-objective selection (Liu et al., 2019).Source: §1 · p. 2

2 Model

• BART is a denoising sequence-to-sequence autoencoder that reconstructs original documents using a bidirectional encoder over corrupted text and a left-to-right autoregressive decoder.Pre-training optimizes the negative log likelihood of the original document.Source: §2 · p. 2

2.1 Architecture

• BART uses the sequence-to-sequence Transformer (Vaswani et al., 2017), with six encoder and decoder layers for the base model and twelve each for large.Following GPT, it replaces ReLU with GeLU (Hendrycks & Gimpel, 2016) and initializes parameters from N(0, 0.02). Relative to BERT, decoder layers add cross-attention over final encoder states and omit BERT’s extra pre-prediction feed-forward network, giving roughly 10% more parameters at equivalent size.Source: §2.1 · p. 2

2.2 Pre-training BART

• BART supports arbitrary document corruption and learns reconstruction through cross-entropy between decoder outputs and the original document.When corruption removes all source information, BART becomes equivalent to a language model.Source: §2.2 · p. 2
• Other composable corruptions replace random tokens with masks, delete tokens, shuffle sentences, or rotate documents to start at a uniformly sampled token.Token masking follows BERT (Devlin et al., 2019), while deletion requires identifying missing input positions. Sentence permutation uses full stops to identify sentences; rotation teaches the model to identify the document’s start.Source: §2.2 · pp. 2–3, Figure 2 · p. 3
• Text infilling replaces each sampled span with a single [MASK] token, teaching BART to predict how many tokens are missing.Span lengths follow a Poisson distribution (λ = 3), with zero-length spans inserting masks. Unlike SpanBERT (Joshi et al., 2019), which uses clamped-geometric lengths and equally long mask sequences, BART uses one mask per span.Source: §2.2 · p. 3

3.1 Sequence Classification Tasks

• Sequence classification feeds identical inputs to BART’s encoder and decoder, then passes the final decoder token’s hidden state to a new multiclass linear classifier.The additional token is placed at the end so its decoder representation can attend to decoder states from the complete input.Source: §3.1 · p. 3

3.2 Token Classification Tasks

• Token classification feeds the complete document into both encoder and decoder, using each word’s top decoder hidden state to classify that token.Applications include answer endpoint classification for SQuAD.Source: §3.2 · p. 3

3.3 Sequence Generation Tasks

• BART directly supports sequence generation by encoding the input sequence and generating outputs autoregressively, including for abstractive question answering and summarization.These tasks copy and manipulate input information, closely relating them to BART’s denoising pretraining objective.Source: §3.3 · p. 3

3.4 Machine Translation

• BART’s entire encoder–decoder serves as a pretrained decoder for translation into English, augmented with a new source encoder learned from bitext.A randomly initialized source encoder replaces BART’s encoder embedding layer, can use a separate vocabulary, and maps foreign words into inputs BART can denoise into English. Training first updates only this encoder, BART’s positional embeddings, and its encoder’s first-layer self-attention input projection matrix, then briefly updates all parameters using output cross-entropy loss throughout.Source: §3.4 · p. 3

4.2 Tasks

• Base-size models evaluate extractive question answering on SQuAD (Rajpurkar et al., 2016) and sentence entailment on MNLI (Williams et al., 2017).Models use six encoder layers, six decoder layers, and hidden size 768. SQuAD predicts answer-span boundaries, while MNLI classifies sentence relations using the EOS-token representation; both pass inputs through encoder and decoder.Source: §4 · p. 3, §4.2 · p. 4

4.1 Comparison Objectives

• Controlled reimplementations compare discriminative and generative objectives; standard encoder–decoder fine-tuning works better for BART, whereas decoder source-prefix conditioning works better for other models.Learning rates and layer-normalization usage are tuned separately for each objective. Source-prefix conditioning applies loss only to the target portion.Source: §4.1 · p. 4

4.2 Tasks

• Generation evaluation reports perplexity across long-form question answering, news summarization, and persona-conditioned dialogue.ELI5 (Fan et al., 2019) generates answers from questions and supporting documents, while ConvAI2 (Dinan et al., 2019) generates responses from dialogue context and a persona. XSum (Narayan et al., 2018) uses highly abstractive news summaries, whereas CNN/DM (Hermann et al., 2015) summaries closely follow source sentences.Source: §4.1 · p. 4, §4.2 · p. 4

4.3 Results

• BART models using text infilling achieve the most consistently strong performance across tasks, except ELI5.Models have comparable sizes and undergo 1M pre-training steps on books and Wikipedia. The reimplemented objectives and BART variants share training data, codebase, and fine-tuning procedures.Source: Table 1 · p. 5, Left-to-right pre-training improves… · p. 5

Left-to-right pre-training improves generation

• A pure language model performs best on ELI5 but worst on SQuAD, demonstrating substantial task dependence.ELI5 has much higher perplexities than the other tasks and is the only generation task where other models outperform BART. This suggests BART is less effective when outputs are loosely constrained by inputs.Source: §4.3 · p. 5, Left-to-right pre-training improves… · p. 5

4.3 Results

• Token corruption is crucial: document rotation and sentence permutation perform poorly alone, while deletion appears better than masking for generation.Successful objectives use token deletion, token masking, or self-attention masks.Source: §4.3 · p. 5

Left-to-right pre-training improves generation

• Left-to-right pre-training benefits generation, while bidirectional context is crucial for SQuAD.Masked and Permuted Language Models lack left-to-right autoregressive pre-training and perform worse on generation; BART achieves similar SQuAD performance with half the bidirectional layers. The simplified Permuted Language Model trails XLNet (Yang et al., 2019), potentially because it omits relative-position embeddings and segment-level recurrence.Source: Left-to-right pre-training improves… · p. 5

5.1 Experimental Setup

• The large model has 12 layers each in its encoder and decoder, hidden size 1024, batch size 8000, and 500000 pre-training steps.Pre-training masks 30% of tokens, permutes all sentences, and uses GPT-2 byte-pair encoding. It uses the same 160Gb corpus as Liu et al. (2019), with dropout disabled for the final 10% of steps.Source: §5.1 · pp. 5–6

5.2 Discriminative Tasks

• BART performs similarly to RoBERTa on SQuAD and GLUE under matched pre-training resources, with only small differences on most tasks.Its generation improvements therefore do not come at the expense of classification performance.Source: Table 2 · p. 6, §5.2 · p. 6

5.3 Generation Tasks

• Roughly 6.0 points on all ROUGE metrics separate BART from the best previous BERT-based XSum system; BART also outperforms all existing CNN/DailyMail work.CNN/DailyMail summaries tend to resemble source sentences, whereas XSum is highly abstractive and extractive models perform poorly.Source: Table 3 · p. 6, §5.3 · p. 6
• Generation fine-tuning uses standard sequence-to-sequence training with label-smoothed cross-entropy (Pereyra et al., 2017).The smoothing parameter is 0.1; generation uses beam size 5, removes duplicate trigrams, and tunes length settings on validation data (Fan et al., 2017).Source: §5.3 · p. 6
• BART outperforms previous work on two automated metrics for CONVAI2 dialogue response generation.Responses are conditioned on both the previous conversation and a textually specified persona.Source: Table 4 · p. 6, §5.3 · p. 6
• 1.2 ROUGE-L above the best previous work is BART’s improvement on ELI5 abstractive question answering.ELI5 tests long, freeform answer generation and remains challenging because questions only weakly specify their answers.Source: Table 5 · p. 7, §5.3 · p. 7

5.4 Translation

• A 6-layer source encoder maps Romanian into representations BART denoises into English on WMT16 Romanian-English augmented with back-translation.Experiments compare fixed and tuned BART against a Transformer baseline (Vaswani et al., 2017). Preliminary results without back-translation were less effective and prone to overfitting.Source: §5.4 · p. 7

6 Qualitative Analysis

• Up to 6 points over the prior state-of-the-art on summarization metrics motivate qualitative analysis beyond automated scores.The analysis examines generated summaries to better understand BART’s performance.Source: §6 · p. 7
• WikiNews examples postdate the pretraining corpus, and first sentences are removed following Narayan et al. (2018) to prevent easy extractive summaries.The article dates were chosen to exclude the described events from the model’s training data.Source: §6 · p. 7
• Fluent, grammatical, highly abstractive, and generally factually accurate summaries suggest that BART pretraining combines strong language understanding and generation.Outputs integrate evidence across the document with background knowledge, including completing names and inferring that PG&E operates in California.Source: §6 · p. 7
• One summary makes a non-trivial inference that fish protect reefs from global warming, but incorrectly adds a source-unsupported claim of publication in Science.This example illustrates that generally accurate summaries can still introduce unsupported factual content.Source: §6 · p. 7

7 Related Work

• GPT models only leftward context; ELMo (Peters et al., 2018) combines directional representations without pretraining their interactions.Radford et al. (2019) showed that very large language models can act as unsupervised multitask models.Source: §7 · p. 7
• BERT (Devlin et al., 2019) learns interactions between left and right context through masked language modelling, but non-autoregressive predictions reduce its effectiveness for generation.Strong performance also follows longer training (Liu et al., 2019), parameter sharing across layers (Lan et al., 2019), and span masking (Joshi et al., 2019).Source: §7 · p. 7
• Unlike UniLM (Dong et al., 2019), BART makes autoregressive predictions and trains its decoder on uncorrupted context, reducing the mismatch between pretraining and generation.MASS reconstructs a masked contiguous span, but disjoint encoder and decoder inputs limit discriminative effectiveness. XL-Net (Yang et al., 2019) predicts masked tokens autoregressively in permuted order using both contexts, whereas BART’s left-to-right decoding matches generation.Source: §7 · pp. 7–8
• BART improves machine-translation decoders, where gains from pretrained representations had been more limited than for encoders (Edunov et al., 2019).The largest previous improvements used source- and target-language pretraining (Lample & Conneau, 2019), requiring pretraining on every language of interest.Source: §7 · p. 8

• BART achieves similar performance to RoBERTa on discriminative tasks while setting new state-of-the-art results on several text generation tasks.Source: §8 · p. 8
• BART pre-training learns to map corrupted documents back to their originals.Source: §8 · p. 8
• Future work should explore new document-corruption methods for pre-training, potentially tailored to specific end tasks.Source: §8 · p. 8