PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Deep Transformer Models for Machine Translation

Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, Lidia S. Chao

arXiv:1906.01787v1cs.CLcs.LG

TL;DR

Deep Transformer encoders are promising but difficult to optimize, and the paper investigates how to make them effective for machine translation. It combines layer-normalization placement with dynamic linear combinations of preceding layers, obtaining stronger or comparable performance than Transformer-Big while reducing model and training costs.

  • Problem

    Simply deepening Transformer has not reliably outperformed wide Transformer-Big models, and deeper networks are difficult to optimize.

  • Method

    The paper uses proper layer-normalization placement and dynamic linear combination of preceding layers to train deep Transformer encoders.

  • Results

    Across three translation tasks, the deep system improves BLEU by 0.4∼2.4 points over Transformer-Big/Base baselines and is 1.6X smaller with 3X less training time.

  • Takeaways & Limitations

    A thin-but-deep Transformer encoder can match or surpass Transformer-Big while using fewer parameters, fewer training epochs, and faster inference.

  • Takeaways & Limitations

    Beyond 30 layers, deeper networks provide no further benefit and consume substantial memory, limiting efficient training.

Abstract

from arXiv · show

Transformer is the state-of-the-art model in recent machine translation evaluations. Two strands of research are promising to improve models of this kind: the first uses wide networks (a.k.a. Transformer-Big) and has been the de facto standard for the development of the Transformer system, and the other uses deeper language representation but faces the difficulty arising from learning deep networks. Here, we continue the line of research on the latter. We claim that a truly deep Transformer model can surpass the Transformer-Big counterpart by 1) proper use of layer normalization and 2) a novel way of passing the combination of previous layers to the next. On WMT'16 English- German, NIST OpenMT'12 Chinese-English and larger WMT'18 Chinese-English tasks, our deep system (30/25-layer encoder) outperforms the shallow Transformer-Big/Base baseline (6-layer encoder) by 0.4-2.4 BLEU points. As another bonus, the deep model is 1.6X smaller in size and 3X faster in training than Transformer-Big.

1 Introduction

The paper addresses whether deep Transformer encoders can improve translation, despite optimization difficulties, using layer normalization and dynamic combinations of preceding layers. The resulting 30-layer encoder surpasses Transformer-Base and Transformer-Big on several translation tasks.

  • Deepening Transformer encoders remains an open question because wide Transformer-Big networks have benefited more than simply adding layers.
  • Deep Transformer training is difficult because gradient vanishing or exploding can make straightforward layer stacking fail.
  • The approach relocates layer normalization to optimize deep encoder networks more smoothly.
  • Dynamic linear combination of layers preserves features from all preceding layers instead of relying only on the immediately preceding residual output.
  • A 30-layer encoder surpasses the deepest previously reported encoder and is presented as the deepest encoder used in neural machine translation.
  • A 1.3∼2.4 BLEU improvement over Transformer-Base and a 0.4∼0.6 BLEU improvement over Transformer-Big are reported across three translation tasks.The deep system uses a 30/25-layer encoder, requires 1.6X fewer model parameters and 3X less training time, and is 10% faster in inference than Transformer-Big.

2 Post-Norm and Pre-Norm Transformer

The section contrasts post-norm and pre-norm residual units and explains why pre-norm is easier to optimize as Transformer depth increases. The analysis connects layer-normalization placement to gradient propagation in deep stacks.

  • Transformer stacks self-attention and feed-forward sub-layers in encoder and decoder layers, with residual connections and layer normalization supporting information flow.
  • Layer normalization reduces sub-layer output variance, addressing hidden-state dynamics that can lengthen convergence time.
  • Post-norm applies layer normalization after residual addition, whereas pre-norm applies it to each sub-layer input.
  • The two normalization choices have comparable BLEU performance with a 6-layer encoder.
  • Pre-norm passes error gradients directly from the top to lower layers through a depth-independent product path.
  • Post-norm gradients pass through layer normalization at every sub-layer, increasing the risk of gradient vanishing or exploding as depth grows.
  • Pre-norm successfully trains a 20-layer encoder on WMT English-German, whereas the post-norm system fails for a deeper encoder.

3 Dynamic Linear Combination of Layers

Dynamic Linear Combination of Layers (DLCL) connects each layer to preceding representations through learned linear combinations, addressing residual networks’ loss of access to distant features. Its learned, flexible connectivity generalizes several existing layer-connection methods.

  • Dynamic Linear Combination of Layers: DLCL directly links each layer to all previous layers, combining their outputs to provide efficient access to lower-level representations in deep stacks.The design is inspired by linear multi-step methods, which reuse previous steps through linear combination.
  • Dynamic Linear Combination of Layers: Each incoming layer receives a learnable scalar weight, allowing layer preferences to vary across the stack and across succeeding layers.For pre-norm Transformer, the combination function merges previously generated values before producing the next layer’s input.
  • Comparison to Existing Methods: DLCL learns connection weights end-to-end and permits an arbitrary past-history window, unlike linear multi-step methods that typically use deterministic weights and limited history.These differences provide more flexible control over model behavior.
  • Comparison to Existing Methods: DLCL generalizes residual, densely residual, multi-layer fusion, and transparent-attention connections by weighting and connecting layers throughout the stack.Standard residual networks are described as a special case, while fusion and transparent attention apply layer weighting only at the topmost layer.

4 Experimental Setup

The experiments evaluate DLCL-based Transformer models on three translation benchmarks using established datasets, BLEU evaluation, repeated runs where feasible, and specified training configurations. The setup includes comparisons with Transformer baselines and prior 16-layer results, while the largest task is run only once.

  • Datasets: Three benchmarks are evaluated: WMT’16 English-German, NIST’12 Chinese-English, and a larger WMT’18 Chinese-English dataset augmented by back-translation.The datasets contain approximately 4.5M, 1.9M, and 16.5M sentence pairs or combined training examples, respectively.
  • Reported Comparisons: Table 1 reports English-German BLEU scores, batch sizes, and approximate training times, with “-deep” denoting the best result as depth changes.The table compares systems under an 8-GPU setting.
  • Evaluation: Evaluation uses tokenized BLEU for English-German and small Chinese-English, detokenized BLEU for large Chinese-English, and mean scores across three random seeds unless otherwise noted.The last five checkpoints are averaged before beam-search decoding.
  • Reported Comparisons: Table 2 compares systems from this work with Bapna et al. (2018) under a 16-layer encoder.The comparison is conducted on WMT’16 English-German translation.
  • Training Configuration: Transformer-Base and Transformer-Big baselines use 100k/300k, 50k/100k, and 200k/500k update steps across the three tasks, respectively.Training uses Adam, warmup schedules, dropout settings, and batches restricted to 4096 tokens per GPU.
  • Model Configuration: DLCL is applied to both encoder and decoder, combining complete-layer outputs; deep models accumulate gradients from smaller batches to address single-GPU memory limits.The setup uses gradient accumulation for models with depth at least 20.

5 Results

Across English-German and Chinese-English tasks, deep pre-norm Transformer variants improve over shallower baselines and can surpass Transformer-Big. DLCL supports deeper encoders while maintaining competitive performance and efficiency.

  • English-German: Vanilla Transformer fails to train at L = 20 in the post-norm setting, whereas post-norm DLCL reaches its best result at L = 25.The authors attribute vanilla Transformer’s failure to vanishing gradients in lower layers.
  • English-German: 1.8 BLEU points: A 20-layer pre-norm Transformer outperforms the 6-layer baseline on English-German.A 30-layer DLCL encoder adds 0.4 BLEU points and exceeds pre-norm Transformer-Big by 0.6 BLEU points.
  • English-German: DLCL outperforms transparent attention with a 16-layer encoder and continues improving when the encoder becomes deeper.Pre-norm Transformer matches transparent attention without requiring its complicated attention design.
  • Depth and speed analyses: BLEU scores are plotted against encoder depth for pre-norm Transformer and DLCL on English-German and Chinese-English tasks.The supplied figure description identifies depth and BLEU as the plotted variables.
  • Chinese-English: About 0.5 BLEU points: On Chinese-English, the 25-layer pre-norm DLCL model exceeds Transformer-Big at equivalent parameter size.Both Transformer and DLCL achieve their best results with a 25-layer encoder on this task.
  • Chinese-English: On the larger Chinese-English task, 25-layer pre-norm DLCL slightly surpasses Transformer-Big, with a larger advantage using 30 layers.This result is reported as evidence that the deep-model advantage is not limited to small datasets.

6 Analysis

The analysis examines how encoder depth affects performance, speed, and design choices. Deep models can match or surpass Transformer-Big, but excessive depth harms efficiency and thin deep encoders outperform deep-and-wide alternatives.

  • Effect of Encoder Depth: At encoder depth 20, both pre-norm Transformer and DLCL achieve performance comparable to Transformer-Big, with DLCL exceeding it beyond 20 layers.Pre-norm Transformer becomes less robust beyond 20 layers, while DLCL continues improving before deeper networks stop helping.
  • Inference Speed: A 30-layer DLCL encoder remains faster than Transformer-Big because encoding runs once and is independent of beam size.Inference speed decreases linearly with encoder depth, while autoregressive decoding remains the main bottleneck.
  • Decoder Depth: Decoder deepening yields only slight BLEU gains, while every two added layers reduce generation speed by approximately 500 tokens per second.The result supports exploring deep encoders rather than deep decoders for NMT.
  • Ablation Study: Learnable layer-combination weights are important: replacing them with constant weights consistently hurts performance, especially for the All-One variant.Removing added layer normalization from the Average model lowers BLEU by 0.28.
  • Model Capacity: Increasing capacity with a big encoder or a 12-layer deep-and-wide Transformer provides smaller gains than the thin deep counterpart in BLEU, parameter size, and training efficiency.The comparison favors depth with the proposed connectivity over simply combining width and depth.
  • Learned Weights: Learned connections are dense in early encoder layers and become sparser with depth, indicating that earlier layers contribute substantially to the deep model.The visualization is from the 30-layer pre-norm DLCL encoder on English-German.

7 Related Work

Prior work addressed deep Transformer optimization and layer connectivity through attention and densely residual connections. This paper extends those directions by learning denser, layer-specific connectivity throughout the network.

  • Deep Models: RNN-based approaches reduced nonlinear transitions to ease optimization, but those techniques are not necessarily directly applicable to Transformer.The related work distinguishes earlier RNN solutions from the Transformer setting.
  • Deep Models: Bapna et al. trained a 16-layer Transformer encoder by attending to the combination of all encoder layers at the decoder.Their work is identified as the most relevant prior approach to deep Transformer training.
  • Densely Residual Connections: Earlier densely residual methods either fixed layer weights or learned distributions, often focusing connections on the topmost layer.These methods include self-attention-based and softmax-normalized learnable weighting schemes.
  • Densely Residual Connections: DLCL adds connectivity throughout the stack and learns denser connections for each layer end to end.Unlike methods focused on the topmost layer, it learns layer interactions during the encoding process.

8 Conclusion

The paper shows that properly normalized deep Transformer encoders can be optimized and that dynamic layer combinations enable a 30-layer system. The resulting thin-but-deep model matches or surpasses Transformer-Big with lower size and training cost.

  • Conclusion: Proper layer normalization and dynamic linear combinations of layers enable successful training of a 30-layer Transformer encoder.The paper identifies this as its central technical outcome.
  • Conclusion: The thin-but-deep encoder can match or surpass Transformer-Big while being 1.6X smaller.The comparison is stated at the model level rather than for a single task or metric.
  • Conclusion: The deep system requires 3X fewer training epochs and is 10% faster for inference.These efficiency results accompany the reported model-size advantage.

A Derivations of Post-Norm Transformer and Pre-Norm Transformer

The derivation compares post-norm and pre-norm residual formulations and explains their different gradient paths. Pre-norm preserves a direct residual route that avoids depth-dependent products in backpropagation.

  • Residual Formulations: Post-norm and pre-norm Transformer are presented as special cases of a general residual unit with different placements of layer normalization.The formulations use Fpost(x)=F(x) and Fpre(x)=F(LN(x)).
  • Gradient Derivation: The derivation applies the chain rule to a stack of L sub-layers, relating the loss gradient to the topmost output and intermediate layer states.E denotes the prediction loss and xL denotes the top-most sub-layer output.
  • Post-Norm Gradient: For post-norm, the number of product terms grows linearly with depth, increasing susceptibility to gradient vanishing or explosion.The derivation attributes the risk to layer-by-layer gradient decomposition.

B Training Hyper-parameters for Deep Models

Hyper-parameters were selected using validation perplexity on English-German translation, comparing standard settings across 6-layer and 20-layer models. Post-norm models failed as depth increased, whereas pre-norm models benefited from larger batches and learning rates, especially in deep networks.

  • Hyper-parameters were selected by measuring validation-set perplexity on the WMT English-German task for 6-layer and 20-layer models.Standard hyperparameters served as baselines for both shallow and deep networks.
  • The baseline configurations used batch size 4096 and 100k updates, with learning rate and warmup settings differing between post-norm and pre-norm Transformer-Base.Post-norm used 7×10−4 learning rate and 4k warmup, while pre-norm used 10−3 and 8k, respectively.
  • Post-norm Transformer failed to train with the deeper network settings, and shallow post-norm models also failed to converge under those settings.The authors attribute this sensitivity to the larger learning rate used for deep-network settings.
  • Pre-norm Transformer benefited from larger batch sizes and learning rates with both 6-layer and 20-layer encoders.The improvement was larger for deep networks than for shallow networks.
Loading 1906.01787v1…