PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Making the Most of BERT in Neural Machine Translation

Jiacheng Yang, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Yong Yu, Weinan Zhang, Lei Li

arXiv:1908.05672v5cs.CLcs.LG

TL;DR

Resource-rich NMT can make fine-tuned BERT forget pre-trained knowledge, limiting the benefit of direct integration. CTNMT combines distillation, dynamic switching, and scheduled learning to retain and use that knowledge. It reports gains across large translation benchmarks, including +1.4 BLEU over prior pre-training-aided NMT on WMT English-German.

  • Problem

    In resource-rich NMT, extensive fine-tuning can cause catastrophic forgetting and diminish the benefits of pre-trained language models.

    Source: §1 · p. 1, §2.1 · p. 3

  • Method

    CTNMT integrates BERT with NMT using asymptotic distillation, a dynamic switching gate, and rate-scheduled updating.

    Source: §1 · p. 2, §2.2 · p. 3, §2.4 · p. 4

  • Results

    +1.4 BLEU over previous pre-training-aided NMT is reported on the WMT English-German benchmark, with gains also shown on other large datasets.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    CTNMT provides an effective, simple, and efficient transfer-learning method for NMT and can also be applied to other NLP tasks.

    Source: §1 · p. 2, §6 · p. 8

    CTNMT is evaluated on the standard Transformer model, although the authors state it is generally applicable to other NMT architectures.

    Source: §2.1 · p. 2

Abstract

from arXiv · show

GPT-2 and BERT demonstrate the effectiveness of using pre-trained language models (LMs) on various natural language processing tasks. However, LM fine-tuning often suffers from catastrophic forgetting when applied to resource-rich tasks. In this work, we introduce a concerted training framework (CTNMT) that is the key to integrate the pre-trained LMs to neural machine translation (NMT). Our proposed CTNMT consists of three techniques: a) asymptotic distillation to ensure that the NMT model can retain the previous pre-trained knowledge; b) a dynamic switching gate to avoid catastrophic forgetting of pre-trained knowledge; and c) a strategy to adjust the learning paces according to a scheduled policy. Our experiments in machine translation show CTNMT gains of up to 3 BLEU score on the WMT14 English-German language pair which even surpasses the previous state-of-the-art pre-training aided NMT by 1.4 BLEU score. While for the large WMT14 English-French task with 40 millions of sentence-pairs, our base model still significantly improves upon the state-of-the-art Transformer big model by more than 1 BLEU score. The code and model can be downloaded from https://github.com/bytedance/neurst/ tree/master/examples/ctnmt.

1 Introduction

CTNMT addresses catastrophic forgetting when integrating BERT into high-resource NMT by combining distillation, adaptive knowledge fusion, and scheduled updating. It improves translation across multiple language pairs and exceeds prior pre-training-aided NMT on WMT English-German.

Motivation

  • Directly using BERT in NMT often yields limited gains on resource-rich benchmarks, including WMT14 English-French.The approach can work in resource-poor scenarios but rarely produces inspiring results with abundant parallel data.Source: §1 · pp. 1–2
  • High-resource NMT requires many updates, which can cause BERT to forget universal pre-training knowledge.NMT combines large training sets with high-capacity Transformer models, unlike tasks requiring only hundreds of updates.Source: §1 · p. 1

Approach

  • CTNMT combines asymptotic distillation, a dynamic switching gate, and rate-scheduled updating to integrate BERT with vanilla NMT.Distillation retains BERT knowledge, switching adaptively fuses BERT and NMT representations, and scheduled updating adjusts learning paces.Source: §1 · p. 2

Results

  • Up to 2.9, 1.3, and 1.6 BLEU gains are reported for English-German, English-French, and English-Chinese, respectively.The results cover three machine-translation language pairs.Source: §1 · p. 2
  • +1.4 BLEU over the previous state-of-the-art pre-training-aided NMT is achieved on WMT English-German.The paper also reports significant improvement over the state-of-the-art Transformer-big model.Source: §1 · p. 2

2 The Proposed CTNMT

CTNMT integrates pre-trained language models into NMT through asymptotic distillation, dynamic switching, and rate-scheduled learning. These components retain pre-trained information while balancing it with NMT training and controlling parameter updates.

Background

  • Directly fine-tuning BERT in resource-rich NMT can cause catastrophic forgetting as large datasets and high-capacity models require many updates.The paper motivates CTNMT by contrasting resource-poor settings, where BERT integration can help, with large NMT benchmarks where gains are limited.Source: §2.1 · p. 3

Asymptotic Distillation

  • Asymptotic distillation regularizes NMT hidden states toward fixed pre-trained language-model representations while retaining the standard NMT objective.The pre-trained model acts as teacher and the NMT encoder as student; the distillation term uses mean-squared error and is balanced with cross-entropy by α.Source: §2.2 · p. 3

Dynamic Switch

  • Dynamic switching uses a sigmoid gate to control element-wise information flow from the pre-trained model and the original NMT model.A gate value of 0 yields the traditional NMT model, whereas 1 corresponds to the fine-tuning approach.Source: §2.3 · p. 4

Rate-scheduled Learning

  • Rate-scheduled learning assigns separate learning rates to pre-trained and NMT parameters to reduce forgetting during fine-tuning.The schedule first increases and then gradually decreases the pre-trained-parameter learning rate, while NMT parameters follow the standard Transformer strategy.Source: §2.4 · pp. 4–5
  • The training schedule first converges NMT parameters, then jointly updates NMT and language-model parameters, and finally refines only NMT parameters.This staged behavior is implemented with a slanted triangular learning-rate policy featuring a short increase and long decay.Source: §2.4 · p. 4

3 Experiments Settings

CTNMT is evaluated on three public WMT translation tasks using cased BLEU, public BERT/GPT-2 models, and Transformer-based NMT settings. The experiments use standard preprocessing and task-specific datasets, with BERTBASE as the default configuration.

Evaluation Tasks

  • CTNMT is evaluated on English-German, English-French, and English-Chinese translation using cased BLEU.The datasets contain 4.5M, 36M, and 2.2M sentence pairs respectively.Source: §3.1 · p. 5

Datasets

  • The English-German experiments use 4.5M WMT14 sentence pairs, with news-test 2012–2013 for validation and news-test 2014 for testing.Source: §3.1 · p. 5
  • The English-French experiments use 36M WMT14 sentence pairs and the same 2012–2013 validation and 2014 test-year split.Source: §3.1 · p. 5
  • English-Chinese training uses 2.2M sentence pairs from WMT18, with WMT17 for development and WMT18 for testing.Source: §3.1 · p. 5

Pre-trained Models

  • The study uses public BERT and GPT-2 models; BERTBASE has 12 layers, 768 hidden units, 12 attention heads, and 110M parameters.GPT-2 is unidirectional and uses only previous context at each time step.Source: §3.2 · p. 5

Training Details

  • For CTNMT, experiments mainly use BERT’s encoder and second-to-last hidden states, with scheduled-learning steps set to T′ = 10,000 and T = 20,000.Dynamic switching combines BERT’s second-to-last layer with the NMT word-embedding layer.Source: §3.2 · p. 5

4 Results and Analysis

CTNMT’s component analyses show that encoder-side BERT integration, asymptotic distillation, dynamic switching, and scheduled learning each address weaknesses of direct fine-tuning. Their combination yields the strongest reported improvements across the evaluated translation tasks.

Overall Results

  • CTNMT strategies improve English-German by almost 2 BLEU, English-French by 1.2 BLEU, and English-Chinese by 1.6 BLEU for base models.Combining the strategies adds roughly 0.5 BLEU over the best single strategy.Source: §4 · p. 6
  • The fused rate-scheduling, dynamic switch, and asymptotic distillation system achieves the best performance in Table 1.Source: §4 · p. 6

Encoder v.s. Decoder

  • BERT representations are most effective on the NMT encoder and less effective on the decoder.BERT’s bidirectional information can create training–inference inconsistencies when used for decoder pre-training.Source: §4.1 · p. 6, §4.2 · p. 6

About asymptotic distillation

  • Asymptotic distillation produces continuous improvements as training data increases, while fine-tuning gains gradually diminish.Source: §4.3 · p. 7

About dynamic switch

  • A 1.3 BLEU gain from average pooling over the baseline increases to 0.9 BLEU beyond average pooling with dynamic switching.Dynamic switching balances BERT features and the NMT representation.Source: §4.4 · p. 7

About rate-scheduled learning

  • Slanted triangular learning rates are more promising for fine-tuning than a fixed small language-model update weight.The policy first increases and then gradually decreases the language-model learning rate.Source: §4.5 · p. 7

About asymptotic distillation

  • The second-to-last BERT layer works significantly better than the last hidden state for asymptotic distillation.The last layer is biased toward masked-language-model and next-sentence-prediction targets.Source: §4.6 · pp. 7–8

5 Related work

Related work applies pre-training and transfer learning across NLP and NMT, including resource-poor settings. CTNMT differs by combining asymptotic distillation, dynamic switching, and rate-scheduled learning to address catastrophic forgetting on large datasets.

Unsupervised pre-training of LMs

  • Unsupervised pre-training and transfer learning improve supervised NLP tasks by reusing representations learned from independently collected data.Prior work spans word, sentence, and contextualized representations, including ELMo and language-model pre-training.Source: §5.1 · p. 8

Pre-training for NMT

  • Pre-trained word embeddings and sequence-to-sequence initialization have improved NMT, particularly for resource-poor languages or domains.Related methods transfer knowledge from resource-rich tasks or initialize sequence-to-sequence model components.Source: §5.2 · p. 8
  • Multilingual NMT improves poorer-resource settings by sharing a source encoder across language pairs and fine-tuning shared parameters.These approaches use different target decoders for different languages.Source: §5.2 · p. 8
  • Earlier methods feed ELMo or BERT representations into the NMT encoder, whereas CTNMT adds asymptotic distillation, dynamic switching, and rate-scheduled learning.The paper presents these additions as addressing catastrophic forgetting and retaining traditional NMT encoder features on tens of millions of training examples.Source: §5.2 · p. 8

6 Conclusion

The paper concludes that CTNMT is a practical transfer-learning method for integrating pre-trained language models into NMT. It recommends encoder integration, CTNMT for catastrophic forgetting, and distillation when computational resources are constrained.

Conclusion

  • CTNMT is an effective, simple, and efficient transfer-learning method for neural machine translation that can also apply to other NLP tasks.The conclusion presents CTNMT as a generalizable approach beyond the evaluated NMT setting.Source: §6 · p. 8
  • Adding pre-trained language models to the NMT encoder is more effective than adding them to the decoder network.This is stated as a practical recommendation for integrating pre-trained models in NMT.Source: §6 · p. 8
  • CTNMT addresses the catastrophic forgetting problem suffered by pre-training for NMT.The conclusion identifies this as a practical effect of employing CTNMT.Source: §6 · p. 9
  • Pre-training distillation is a good choice for computationally resource-constrained scenarios.The conclusion links this recommendation to CTNMT's reported performance.Source: §6 · p. 9
  • 1.4 BLEU is the reported margin by which CTNMT surpasses previous pre-training approaches on the WMT English-German benchmark.The conclusion also states that CTNMT achieves remarkable performance on two other large datasets.Source: §6 · p. 9

Related papers

Loading 1908.05672v5…