Source-linked AI summary
Training Deeper Neural Machine Translation Models with Transparent Attention
Ankur Bapna, Mia Xu Chen, Orhan Firat, Yuan Cao, Yonghui Wu
TL;DR
Although modern neural machine translation models have substantial capacity, their depth remains limited and deeper encoders are difficult to optimize. The paper proposes transparent attention, which regulates gradient flow through encoder depth and enables deeper Transformer and RNMT+ models. Transparent attention yields consistent translation-quality gains across WMT’14 English-German and WMT’15 Czech-English, while the study remains focused on deeper narrow encoders rather than extremely deep and wide models.
Problem
Current neural machine translation models remain shallow relative to convolutional models, while deeper encoders face computational and trainability challenges.
Method
Transparent attention uses trainable weighted connections to disperse error signals across encoder depth and time, adjusting gradient flow across layers.
Results
Transparent attention enables encoders up to 20 layers and improves performance by more than 0.7 BLEU points on both translation datasets.
Takeaways & Limitations
Deeper architectures offer potential translation-quality improvements despite optimization challenges.
Takeaways & Limitations
The study explores deeper encoders for narrow models and leaves extremely deep and wide models for future work.
Abstract
from arXiv · showhide
While current state-of-the-art NMT models, such as RNN seq2seq and Transformers, possess a large number of parameters, they are still shallow in comparison to convolutional models used for both text and vision applications. In this work we attempt to train significantly (2-3x) deeper Transformer and Bi-RNN encoders for machine translation. We propose a simple modification to the attention mechanism that eases the optimization of deeper models, and results in consistent gains of 0.7-1.1 BLEU on the benchmark WMT'14 English-German and WMT'15 Czech-English tasks for both architectures.
1 Introduction
Neural machine translation has improved through new architectures and larger capacities, but deeper models remain difficult to train because of optimization constraints. This study addresses that gap by training deep Transformer and LSTM encoders and modifying attention to support optimization.
- Deeper models can extract more expressive features, but research on them is constrained by computation and trainability problems such as vanishing or exploding gradients.These issues remain under-explored for Transformers or LSTM seq2seq models deeper than 8 layers.
- The study trains deep encoders for Transformer and LSTM-based translation models, where vanilla Transformers fail completely as encoder depth increases.The authors also observe sub-optimal performance for deeper LSTM models, which they associate with trainability issues.
- The proposed attention enhancement eases optimization of deeper models and produces consistent gains on WMT’14 English-German and WMT’15 Czech-English tasks.
2 Transparent Attention
Transparent attention lets the decoder combine outputs from all encoder layers, improving gradient flow and enabling deeper Transformer encoders to train. Experiments compare deep Transformer and RNMT+ models using gradient diagnostics and BLEU evaluation.
- 2 Transparent Attention: Deeper Transformer encoders fail to train, with the 20-layer model showing a persistently low grad-norm ratio compared with the 6-layer model.The ratio remains below 1.0 for both models, but the issue is less severe in the shallow model.
- 2 Transparent Attention: Deep RNMT+ encoders reach the 6-layer model’s performance, with similar gradient dynamics across shallow and deep models.This contrasts with the unstable training observed as Transformer depth increases.
- 2 Transparent Attention: Removing residual connections severely harms Transformer training, while 6-layer RNMT+ still converges with only slight quality degradation and deeper RNMT+ models fail.These baselines indicate that gradient-flow regulation is important for optimizing deeper encoders.
- 2 Transparent Attention: Transparent attention lets the decoder attend to weighted combinations of all encoder-layer outputs rather than only the top layer.The mechanism uses trainable weights to adjust gradient flow across encoder depth during training.
3 Results and Analysis
Transparent attention improves most Transformer experiments, with the largest gains in deeper models. Its learned layer weights also shift toward deeper encoder layers during training.
- More than 0.7 BLEU improvement on both datasets was achieved by training transparent-attention Transformers with up to 20 encoder layers, whereas baseline Transformers failed at 12 layers or deeper.These deeper models also matched or exceeded Transformer Big with less than half its model capacity.
- Transparent attention improved performance in most Transformer experiments, with the strongest gains occurring for deeper models.
- The learned attention weights initially emphasize the lowest embedding layer, then shift toward layers 16 and 20 after those lower layers converge.Weights for the top few layers remain comparable at convergence, suggesting a possible encoder-feature ensembling effect.
4 Conclusions and Future Work
The study finds that deeper encoders create optimization challenges associated with hindered gradient flow, which transparent attention helps address. It reports gains across two translation tasks but limits its exploration to narrow deep encoders.
- Transformer models became extremely difficult to train beyond 12 encoder layers, while deeper RNMT+ models trained without large performance improvements.
- Transparent attention enabled successful training of deeper Transformer and RNMT+ models, producing consistent translation-quality gains on WMT’14 En→De and WMT’15 Cs→En.
- The study explores deeper encoders for narrow models and leaves extremely deep and wide architectures for future work.