PaperlayerView exact revision on arXiv

Source-linked AI summary

Distilling Knowledge Learned in BERT for Text Generation

Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, Jingjing Liu

arXiv:1911.03829v3cs.CLcs.LG

TL;DR

Using BERT for language generation remains an open question, in part because its masked-language-modeling objective is not autoregressive. The paper finetunes BERT with Conditional Masked Language Modeling and distills its bidirectional predictions into Seq2Seq models. It reports improvements over strong Transformer baselines across multiple generation tasks and new state of the art on two translation datasets.

  • Problem

    How to use BERT for language generation remains an open question, despite its success in language understanding.

    Source: §1 · p. 1

  • Method

    The paper finetunes BERT with Conditional Masked Language Modeling and distills its bidirectional word-probability predictions into a Seq2Seq student.

    Source: §1 · p. 1, §3.3 · pp. 4–5

  • Results

    The method improves over strong Transformer baselines on multiple generation tasks and achieves new state of the art on IWSLT14 German-English and IWSLT15 English-Vietnamese.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    BERT-derived soft supervision can guide conventional autoregressive models with bidirectional context while leaving the teacher out of inference.

    Source: §1 · p. 1

    The summarization test split contains only 1,951 noisy examples, including corrupted inputs and summaries.

    Source: §4.4 · p. 7

Abstract

from arXiv · show

Large-scale pre-trained language model such as BERT has achieved great success in language understanding tasks. However, it remains an open question how to utilize BERT for language generation. In this paper, we present a novel approach, Conditional Masked Language Modeling (C-MLM), to enable the finetuning of BERT on target generation tasks. The finetuned BERT (teacher) is exploited as extra supervision to improve conventional Seq2Seq models (student) for better text generation performance. By leveraging BERT's idiosyncratic bidirectional nature, distilling knowledge learned in BERT can encourage auto-regressive Seq2Seq models to plan ahead, imposing global sequence-level supervision for coherent text generation. Experiments show that the proposed approach significantly outperforms strong Transformer baselines on multiple language generation tasks such as machine translation and text summarization. Our proposed model also achieves new state of the art on IWSLT German-English and English-Vietnamese MT datasets. Code is available at https://github.com/ChenRocks/Distill-BERT-Textgen.

1 Introduction

The paper addresses the open question of using BERT for text generation by distilling a target-finetuned BERT into conventional generation models. Its bidirectional predictions provide future-context guidance, while the approach remains modular across model architectures.

1 Introduction

  • BERT’s masked-language-modeling objective is not autoregressive, leaving its direct use for autoregressive generation an open challenge.The paper studies how to use BERT to improve generation conditioned on input text.Source: §1 · p. 1
  • Conditional Masked Language Modeling (C-MLM) finetunes BERT on a target generation dataset, then uses it as a teacher whose word-probability outputs train a Seq2Seq student.At inference, BERT is not required, so decoding speed is that of the underlying student model.Source: §1 · pp. 1–2
  • BERT’s future-context information regularizes one-step-ahead training, encouraging generation models to account for long-term dependencies and global coherence.The paper contrasts this with maximum-likelihood training, which supplies no explicit signal for global planning.Source: §1 · p. 2
  • The approach is modular across text-generation architectures and model sizes, unlike initializing a Seq2Seq encoder or decoder with BERT parameters.The authors report improvements over strong Transformer baselines across machine translation and summarization, with new state of the art on IWSLT14 German-English and IWSLT15 English-Vietnamese.Source: §1 · p. 2

2 Related Work

Prior work explored BERT and related pretrained models for generation, while this paper uses BERT as a source of training-time regularization rather than embedding its parameters in Seq2Seq. It also distinguishes this use of distillation from model compression.

BERT for Text Generation

  • Prior BERT-generation work explored unsupervised generation, BERT-based summarization, and pretraining; concurrent studies also examined conditional MLM and BERT finetuning for translation.Examples include Wang and Cho (2019), Zhang et al. (2019a), Song et al. (2019), Ghazvininejad et al. (2019), and Yang et al. (2019).Source: §2 · p. 2
  • Rather than using BERT parameters in Seq2Seq, this approach uses BERT to regularize maximum-likelihood training by injecting future information into present-word prediction.The paper describes the broader space of applying pretrained language models to generation as relatively unexplored.Source: §2 · p. 2

Right-to-Left Generation; Knowledge Distillation

  • Unlike prior knowledge-distillation work focused mostly on compression, this method uses a BERT teacher not designed for the student’s generation task; it shares an intuition with reverse-model approaches but uses a pretrained language model.The related reverse-generation examples include Liu et al. (2016), Serdyuk et al. (2018), and Zhang et al. (2019b); prior distillation examples include Bucilu et al. (2006), Hinton et al. (2015), and Kim and Rush (2016).Source: §2 · p. 3

3 Approach

The approach adapts BERT to target generation with Conditional Masked Language Modeling, then distills its bidirectional word distributions into a Seq2Seq model. The student combines this soft supervision with ground-truth labels, retaining an autoregressive generation model.

3.1 Sequence-to-Sequence Learning

  • Seq2Seq models estimate the likelihood of an output sequence conditioned on an input and typically train by maximum likelihood, predicting each token from the source and prior target tokens.The framework can use an attention-based RNN, Transformer, or another neural sequence-generation model.Source: §3.1 · p. 3, §3.2 · p. 3

3.2 Finetune BERT with Conditional MLM

  • C-MLM finetunes BERT for a target task by concatenating input and output, then masking 15% of tokens only in the output for prediction.Because C-MLM is not feasible for direct sequential generation, the paper uses distillation to transfer its learned knowledge to Seq2Seq.Source: §3.2 · p. 4

3.3 Knowledge Distillation for Generation

  • The finetuned BERT teacher provides word-probability distributions informed by left and right context, which the autoregressive Seq2Seq student learns to match as future-context guidance.The paper interprets this additional loss as sequence-level guidance that encourages planning for future context.Source: §3.3 · pp. 4–5
  • The student combines BERT’s soft estimates with ground-truth labels using a weighted objective, and the method applies to Seq2Seq models trained to estimate word-level probabilities.The hyperparameter α tunes the relative importance of the two training targets.Source: §3.3 · pp. 4–5

4 Experiments

The experiments evaluate machine translation and abstractive text summarization.

4.1 Datasets

  • IWSLT15 En-Vi (113k), IWSLT14 De-En (160k), WMT14 En-De (4.5M), and Gigaword are evaluated; ROUGE is reported on test-dev and the official test split.The authors observed severe validation/test distribution mismatch for Gigaword and sampled dev and test-dev splits from validation data.Source: §4.1 · p. 5

4.2 Implementation Details

  • All tasks use a 6-layer base Transformer; BERT logits are precomputed for distillation with top-K set to 8, and inference uses beam search with size 4.Hyperparameters are tuned on development data, and the proposed objective is trained longer because it continues improving after the baseline begins to plateau.Source: §4.2 · pp. 5–6

4.3 Results on Machine Translation

  • 1.54 BLEU points over Transformer on IWSLT14 De-En reaches new state of the art; the method improves Transformer baselines across three MT datasets.On En-Vi, the best model also outperforms Seq2Seq-OT and reported ELMo/CVT results; on WMT14 En-De, it improves the well-tuned Transformer baseline.Source: Table 1 · p. 6, Table 2 · p. 6, Table 3 · p. 6

4.4 Results on Abstractive Summarization

  • The method improves all reported ROUGE metrics on test-dev and is comparable across metrics to more complex summarization-specific systems on the official test split.The paper notes that the official test split contains only 1,951 noisy examples.Source: Table 4 · p. 7, Table 5 · p. 7, §4.4 · p. 7

4.5 Ablation Study

  • A smaller BERT teacher works well, full BERT improves further, and the left-to-right teacher slightly hurts; the authors suggest bidirectional knowledge may matter more than extra parameters.They also report that pretrained weights remain useful for more stable C-MLM training.Source: §4.5 · p. 7, Table 6 · p. 7

4.6 Generation for Different Lengths

  • BLEU gains are more consistent on longer outputs: De-En improves on longer pairs, WMT En-De gains on longer cases despite weaker very-short outputs, and En-Vi follows the trend above length 24.The length analysis is on development data; Gigaword is omitted because almost all summaries are short.Source: §4.6 · pp. 7–8, Figure 2 · p. 8, Figure 3 · p. 8

4.7 Qualitative Examples

  • Qualitative examples show BERT soft targets using right context to downweight locally plausible errors, helping the student produce more coherent translations despite being unable to look ahead at test time.The paper illustrates a baseline failure to recover from “with” and “of.”Source: Table 7 · p. 8, §4.7 · p. 8

5 Conclusion

The paper presents a generic approach for using pretrained language models to improve text generation without parameter sharing, feature extraction, or auxiliary tasks. It reports gains over strong Transformer baselines across multiple tasks, with new state of the art on some translation tasks, and proposes multimodal input as future work.

5 Conclusion

  • The paper proposes a generic approach to improving text generation with pretrained language models.The conclusion frames multimodal input, including image captioning, as future work.Source: §5 · pp. 8–9
  • The approach improves machine translation and abstractive summarization over strong Transformer baselines, achieving new state of the art on some translation tasks.The paper identifies extending Conditional MLM to multimodal input such as image captioning as future work.Source: §5 · p. 9
  • Conditional MLM adapts pretrained language models to supervised sequence-to-sequence tasks, with the teacher influencing the student through soft-label distributions.The approach does not require explicit parameter sharing, feature extraction, or auxiliary tasks.Source: §5 · pp. 8–9

A Implementaion Details and Hyper-parameter Values

The implementation precomputes teacher logits using repeated training pairs and circular masks, then uses task-specific training settings. Experiments ran on Titan RTX or V100 GPUs, with development-set-only hyperparameter tuning.

A Implementaion Details and Hyper-parameter Values

  • Experiments ran on one Titan RTX or V100 GPU, except WMT English-German training used four V100s; hyperparameters were tuned on development data only.Large batches that exceeded GPU memory used gradient accumulation; batch sizes count tokens.Source: A Implementaion Details and… · p. 12
  • Teacher logits were precomputed once per training set after repeating each pair seven times and applying a circular mask to approximate BERT’s 15% masking rate.The masked positions provide teacher soft probabilities for output tokens; Figure 4 marks the [MASK] positions in gray.Source: A Implementaion Details and… · p. 12, Figure 4 · p. 12
  • IWSLT German-English and English-Vietnamese proposed models trained for 100k steps with 8k warmup steps and 6k-token batches, using α = 0.5 and T = 10 for De-En and α = 0.1 and T = 5 for En-Vi.Gigaword’s proposed model trained for 70k steps with 4k warmup steps, 36k-token batches, α = 0.1, and T = 10.Source: A Implementaion Details and… · p. 12
  • WMT English-German used baseline Transformer initialization for the proposed model, which continued training for 50k steps with 4k warmup steps and 64k-token batches.The proposed model used η = 2, α = 0.1, and T = 5.Source: A Implementaion Details and… · p. 12

B Additional Generation Examples

The additional examples describe more coherent generations from the proposed model than from the baseline Transformer. The examples cover German-English translation and Gigaword summarization.

B Additional Generation Examples

  • Qualitative examples report that Transformer + BERT Teacher generates more coherent sentences than the baseline Transformer.The examples include Gigaword summaries and additional English-German generations.Source: B Additional Generation Examples · p. 12
  • The German-English example contrasts a baseline mistake that fails to recover from a future phrase with the proposed model’s correct word choice and more coherent sentence.Parenthetical numbers in Table 8 are sentence-level BLEU scores.Source: Table 8 · p. 13
  • In Gigaword examples, the baseline suffers from early mistakes, while the proposed model generates more coherent summaries.Source: Table 9 · p. 13

Related papers

Loading 1911.03829v3…