Towards Making the Most of BERT in Neural Machine Translation

Jiacheng Yang; Mingxuan Wang; Hao Zhou; Chengqi Zhao; Yong Yu; Weinan Zhang; Lei Li

Source-linked AI summary

Towards Making the Most of BERT in Neural Machine Translation

Jiacheng Yang, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Yong Yu, Weinan Zhang, Lei Li

arXiv:1908.05672v5Aug 15, 2019cs.CLcs.LG

TL;DR

Resource-rich NMT can make fine-tuned BERT forget pre-trained knowledge, limiting the benefit of direct integration. CTNMT combines distillation, dynamic switching, and scheduled learning to retain and use that knowledge. It reports gains across large translation benchmarks, including +1.4 BLEU over prior pre-training-aided NMT on WMT English-German.

Problem
In resource-rich NMT, extensive fine-tuning can cause catastrophic forgetting and diminish the benefits of pre-trained language models.
Method
CTNMT integrates BERT with NMT using asymptotic distillation, a dynamic switching gate, and rate-scheduled updating.
Results
+1.4 BLEU over previous pre-training-aided NMT is reported on the WMT English-German benchmark, with gains also shown on other large datasets.
Takeaways & Limitations
CTNMT provides an effective, simple, and efficient transfer-learning method for NMT and can also be applied to other NLP tasks.
Takeaways & Limitations
CTNMT is evaluated on the standard Transformer model, although the authors state it is generally applicable to other NMT architectures.

Abstract

from arXiv · show

GPT-2 and BERT demonstrate the effectiveness of using pre-trained language models (LMs) on various natural language processing tasks. However, LM fine-tuning often suffers from catastrophic forgetting when applied to resource-rich tasks. In this work, we introduce a concerted training framework (CTNMT) that is the key to integrate the pre-trained LMs to neural machine translation (NMT). Our proposed CTNMT consists of three techniques: a) asymptotic distillation to ensure that the NMT model can retain the previous pre-trained knowledge; b) a dynamic switching gate to avoid catastrophic forgetting of pre-trained knowledge; and c) a strategy to adjust the learning paces according to a scheduled policy. Our experiments in machine translation show CTNMT gains of up to 3 BLEU score on the WMT14 English-German language pair which even surpasses the previous state-of-the-art pre-training aided NMT by 1.4 BLEU score. While for the large WMT14 English-French task with 40 millions of sentence-pairs, our base model still significantly improves upon the state-of-the-art Transformer big model by more than 1 BLEU score. The code and model can be downloaded from https://github.com/bytedance/neurst/ tree/master/examples/ctnmt.

1 Introduction

CTNMT addresses catastrophic forgetting when integrating BERT into high-resource NMT by combining distillation, adaptive knowledge fusion, and scheduled updating. It improves translation across multiple language pairs and exceeds prior pre-training-aided NMT on WMT English-German.

Motivation: Directly using BERT in NMT often yields limited gains on resource-rich benchmarks, including WMT14 English-French.The approach can work in resource-poor scenarios but rarely produces inspiring results with abundant parallel data.
Motivation: High-resource NMT requires many updates, which can cause BERT to forget universal pre-training knowledge.NMT combines large training sets with high-capacity Transformer models, unlike tasks requiring only hundreds of updates.
Approach: CTNMT combines asymptotic distillation, a dynamic switching gate, and rate-scheduled updating to integrate BERT with vanilla NMT.Distillation retains BERT knowledge, switching adaptively fuses BERT and NMT representations, and scheduled updating adjusts learning paces.
Results: Up to 2.9, 1.3, and 1.6 BLEU gains are reported for English-German, English-French, and English-Chinese, respectively.The results cover three machine-translation language pairs.
Results: +1.4 BLEU over the previous state-of-the-art pre-training-aided NMT is achieved on WMT English-German.The paper also reports significant improvement over the state-of-the-art Transformer-big model.

2 The Proposed CTNMT

CTNMT integrates pre-trained language models into NMT through asymptotic distillation, dynamic switching, and rate-scheduled learning. These components retain pre-trained information while balancing it with NMT training and controlling parameter updates.

Background: Directly fine-tuning BERT in resource-rich NMT can cause catastrophic forgetting as large datasets and high-capacity models require many updates.The paper motivates CTNMT by contrasting resource-poor settings, where BERT integration can help, with large NMT benchmarks where gains are limited.
Asymptotic Distillation: Asymptotic distillation regularizes NMT hidden states toward fixed pre-trained language-model representations while retaining the standard NMT objective.The pre-trained model acts as teacher and the NMT encoder as student; the distillation term uses mean-squared error and is balanced with cross-entropy by α.
Dynamic Switch: Dynamic switching uses a sigmoid gate to control element-wise information flow from the pre-trained model and the original NMT model.A gate value of 0 yields the traditional NMT model, whereas 1 corresponds to the fine-tuning approach.
Rate-scheduled Learning: Rate-scheduled learning assigns separate learning rates to pre-trained and NMT parameters to reduce forgetting during fine-tuning.The schedule first increases and then gradually decreases the pre-trained-parameter learning rate, while NMT parameters follow the standard Transformer strategy.
Rate-scheduled Learning: The training schedule first converges NMT parameters, then jointly updates NMT and language-model parameters, and finally refines only NMT parameters.This staged behavior is implemented with a slanted triangular learning-rate policy featuring a short increase and long decay.

3 Experiments Settings

CTNMT is evaluated on three public WMT translation tasks using cased BLEU, public BERT/GPT-2 models, and Transformer-based NMT settings. The experiments use standard preprocessing and task-specific datasets, with BERTBASE as the default configuration.

Evaluation Tasks: CTNMT is evaluated on English-German, English-French, and English-Chinese translation using cased BLEU.The datasets contain 4.5M, 36M, and 2.2M sentence pairs respectively.
Datasets: The English-German experiments use 4.5M WMT14 sentence pairs, with news-test 2012–2013 for validation and news-test 2014 for testing.
Datasets: The English-French experiments use 36M WMT14 sentence pairs and the same 2012–2013 validation and 2014 test-year split.
Datasets: English-Chinese training uses 2.2M sentence pairs from WMT18, with WMT17 for development and WMT18 for testing.
Pre-trained Models: The study uses public BERT and GPT-2 models; BERTBASE has 12 layers, 768 hidden units, 12 attention heads, and 110M parameters.GPT-2 is unidirectional and uses only previous context at each time step.
Training Details: For CTNMT, experiments mainly use BERT’s encoder and second-to-last hidden states, with scheduled-learning steps set to T′ = 10,000 and T = 20,000.Dynamic switching combines BERT’s second-to-last layer with the NMT word-embedding layer.

4 Results and Analysis

CTNMT’s component analyses show that encoder-side BERT integration, asymptotic distillation, dynamic switching, and scheduled learning each address weaknesses of direct fine-tuning. Their combination yields the strongest reported improvements across the evaluated translation tasks.

Overall Results: CTNMT strategies improve English-German by almost 2 BLEU, English-French by 1.2 BLEU, and English-Chinese by 1.6 BLEU for base models.Combining the strategies adds roughly 0.5 BLEU over the best single strategy.
Overall Results: The fused rate-scheduling, dynamic switch, and asymptotic distillation system achieves the best performance in Table 1.
Encoder v.s. Decoder: BERT representations are most effective on the NMT encoder and less effective on the decoder.BERT’s bidirectional information can create training–inference inconsistencies when used for decoder pre-training.
About asymptotic distillation: Asymptotic distillation produces continuous improvements as training data increases, while fine-tuning gains gradually diminish.
About dynamic switch: A 1.3 BLEU gain from average pooling over the baseline increases to 0.9 BLEU beyond average pooling with dynamic switching.Dynamic switching balances BERT features and the NMT representation.
About rate-scheduled learning: Slanted triangular learning rates are more promising for fine-tuning than a fixed small language-model update weight.The policy first increases and then gradually decreases the language-model learning rate.
About asymptotic distillation: The second-to-last BERT layer works significantly better than the last hidden state for asymptotic distillation.The last layer is biased toward masked-language-model and next-sentence-prediction targets.

5 Related work

Related work applies pre-training and transfer learning across NLP and NMT, including resource-poor settings. CTNMT differs by combining asymptotic distillation, dynamic switching, and rate-scheduled learning to address catastrophic forgetting on large datasets.

Unsupervised pre-training of LMs: Unsupervised pre-training and transfer learning improve supervised NLP tasks by reusing representations learned from independently collected data.Prior work spans word, sentence, and contextualized representations, including ELMo and language-model pre-training.
Pre-training for NMT: Pre-trained word embeddings and sequence-to-sequence initialization have improved NMT, particularly for resource-poor languages or domains.Related methods transfer knowledge from resource-rich tasks or initialize sequence-to-sequence model components.
Pre-training for NMT: Multilingual NMT improves poorer-resource settings by sharing a source encoder across language pairs and fine-tuning shared parameters.These approaches use different target decoders for different languages.
Pre-training for NMT: Earlier methods feed ELMo or BERT representations into the NMT encoder, whereas CTNMT adds asymptotic distillation, dynamic switching, and rate-scheduled learning.The paper presents these additions as addressing catastrophic forgetting and retaining traditional NMT encoder features on tens of millions of training examples.

6 Conclusion

The paper concludes that CTNMT is a practical transfer-learning method for integrating pre-trained language models into NMT. It recommends encoder integration, CTNMT for catastrophic forgetting, and distillation when computational resources are constrained.

Conclusion: CTNMT is an effective, simple, and efficient transfer-learning method for neural machine translation that can also apply to other NLP tasks.The conclusion presents CTNMT as a generalizable approach beyond the evaluated NMT setting.
Conclusion: Adding pre-trained language models to the NMT encoder is more effective than adding them to the decoder network.This is stated as a practical recommendation for integrating pre-trained models in NMT.
Conclusion: CTNMT addresses the catastrophic forgetting problem suffered by pre-training for NMT.The conclusion identifies this as a practical effect of employing CTNMT.
Conclusion: Pre-training distillation is a good choice for computationally resource-constrained scenarios.The conclusion links this recommendation to CTNMT's reported performance.
Conclusion: 1.4 BLEU is the reported margin by which CTNMT surpasses previous pre-training approaches on the WMT English-German benchmark.The conclusion also states that CTNMT achieves remarkable performance on two other large datasets.