PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformers without Tears: Improving the Normalization of Self-Attention

Toan Q. Nguyen, Julian Salazar

arXiv:1910.05895v2cs.CLcs.LGstat.ML

TL;DR

Transformer training is sensitive to normalization placement, initialization scale, and embedding normalization. The paper evaluates PreNorm, ScaleNorm, FixNorm, and smaller initializations, finding consistent low-resource gains but a high-resource performance trade-off for PreNorm.

  • Problem

    The paper investigates why Transformer normalization choices affect convergence and performance, including convergence failures associated with post-norm training.

  • Method

    The paper evaluates PreNorm, smaller initializations, ScaleNorm, and FixNorm as normalization-centric changes to Transformer training.

  • Results

    +1.1 BLEU is the average improvement from the combined changes across five low-resource translation pairs, while PreNorm degrades base Transformer performance on WMT'14 English-German.

  • Takeaways & Limitations

    PreNorm is more stable and competent in low-resource NMT, whereas ScaleNorm and FixNorm remain competitive in high-resource settings and train faster.

  • Takeaways & Limitations

    The paper leaves the relationship between PostNorm and PreNorm with other optimizers and normalization-free initialization methods for future investigation.

Abstract

from arXiv · show

We evaluate three simple, normalization-centric changes to improve Transformer training. First, we show that pre-norm residual connections (PreNorm) and smaller initializations enable warmup-free, validation-based training with large learning rates. Second, we propose $\ell_2$ normalization with a single scale parameter (ScaleNorm) for faster training and better performance. Finally, we reaffirm the effectiveness of normalizing word embeddings to a fixed length (FixNorm). On five low-resource translation pairs from TED Talks-based corpora, these changes always converge, giving an average +1.1 BLEU over state-of-the-art bilingual baselines and a new 32.8 BLEU on IWSLT'15 English-Vietnamese. We observe sharper performance curves, more consistent gradient norms, and a linear relationship between activation scaling and decoder depth. Surprisingly, in the high-resource setting (WMT'14 English-German), ScaleNorm and FixNorm remain competitive but PreNorm degrades performance.

1 Introduction

The paper examines normalization choices as major determinants of Transformer convergence and performance. PreNorm, ScaleNorm, FixNorm, and smaller initializations improve training behavior and low-resource translation, while PreNorm degrades performance in high-resource English-German.

  • PreNorm enables warmup-free, validation-based training with large learning rates, including for small batches.The paper contrasts this with prior large-batch scaling results.
  • PreNorm degrades base Transformer performance on WMT'14 English-German despite its benefits in low-resource settings.The paper reports this as a high-resource exception to its low-resource findings.
  • ScaleNorm replaces LayerNorm with scaled ℓ2 normalization using a single learned scale, while FixNorm normalizes word embeddings to a fixed length.Together, these changes target simpler normalization and fixed embedding lengths.
  • +1.1 BLEU is the average improvement from applying PreNorm, FixNorm, and ScaleNorm across five low-resource translation pairs.Each addition contributes at least +0.3 BLEU, and the models attain 32.8 BLEU on IWSLT'15 English-Vietnamese.

2 Background

The paper motivates normalization changes through residual-path stability, initialization scale, and simpler activation normalization. It then revisits learning-rate schedules and embedding normalization as complementary design choices.

  • Identity mappings for transformers: PreNorm preserves an identity residual path, while post-norm LayerNorms can act like residual scaling factors different from 1.The paper conjectures that this contributes to convergence failures and motivates warmup for post-norm training.
  • Identity mappings for transformers: Post-norm places LayerNorm after each sublayer and residual addition, whereas PreNorm places it immediately before the sublayer.The paper compares these placements as alternative residual-unit designs.
  • Weight initialization: PostNorm sometimes fails to converge, especially in low-resource settings, even with many warmup steps; the paper attributes this partly to overly large Xavier-initialized weights.The proposed mitigation is to reduce attention-layer initialization scale with SmallInit.
  • Scaled ℓ2 normalization and FixNorm: ScaleNorm projects activation vectors onto a hypersphere with learned radius g, replacing LayerNorm’s 2d scale-and-shift parameters with one learned scalar.The design encodes a global activation scale for each sublayer.
  • Scaled ℓ2 normalization and FixNorm: FixNorm applies ℓ2 normalization with fixed g at the final linear layer to increase angular differences and aid rare-word translation.Making g learnable allows joint use of ScaleNorm and FixNorm.
  • Learning rates: The learning-rate experiments revisit warmup-free training, larger learning rates, and validation-based decay under the proposed normalization changes.The paper examines whether these modifications support higher learning rates with much smaller batches.

3 Experiments and results

Across low-resource translation experiments, smaller initialization, PreNorm, FixNorm, and ScaleNorm improve convergence, BLEU, or training speed, while high-resource results reveal a trade-off for PreNorm.

  • Experimental setup: Five low-resource translation pairs were evaluated from TED Talks and IWSLT'15 corpora.The en→vi pair comes from IWSLT 2015; the remaining pairs come from TED Talks.
  • 3.1 Large vs. small initialization: With default initialization, PostNorm fails to converge on en→vi even after 16k warmup steps, reaching only 5.76 BLEU.Smaller initialization restores PostNorm convergence, while PreNorm works across all tested setups.
  • 3.2 Scaled ℓ2 normalization and FIXNORM: ScaleNorm requires O(3d) operations versus LayerNorm's O(7d) and makes training around 5% faster in the implementation.ScaleNorm significantly improves LayerNorm on gl→en and sk→en and performs comparably on the other tasks.
  • 3.3 Learning rates: PreNorm permits a maximum learning rate of 10^-3 with small 4k-token batches, although doubling the learning rate does not improve performance.Nowarmup performs comparably to the tested schedulers except on gl→en, where warmup remains useful in the lowest-resource setting.
  • 3.4 High-resource setting: On WMT'14 English-German, ScaleNorm and FixNorm match or exceed LayerNorm, but PostNorm performs notably better than PreNorm.The authors speculate about identity residual networks and state that further study is required.

4 Analysis

The analysis compares normalization choices through BLEU curves, gradient norms, learned scaling values, and label-smoothing effects. PRENORM trains faster and more smoothly, while SCALENORM and FIXNORM influence performance and activation scaling in depth-dependent ways.

  • Performance curves: PRENORM learns faster than POSTNORM and outperforms it throughout en→vi training.With warmup, SCALENORM initially learns more slowly than without warmup, but the variants converge at about the same rate.
  • Gradient norms: POSTNORM produces noisy gradients with sharp spikes, whereas PRENORM produces fewer, smaller gradient fluctuations even without warmup.LAYERNORM has lower global gradient norms than SCALENORM + FIXNORM but includes more normalization-related gradient components.
  • Normalization comparisons: SCALENORM performs comparably or better than RMSNORM across the tested normalization configurations.A fixed-g SCALENORM model is generally comparable to learnable-g SCALENORM, but performs much worse at higher learning rates on ar→en, en→he, and en→vi.
  • Activation scaling: Learned g values generally increase with depth for attention and feedforward-related sublayers, with the clearest linear scaling in the decoder toward the output layer.Decoder-encoder sublayers are the main exception to the positive depth correlation.
  • Activation scaling: Removing label smoothing lowers g in later decoder layers but sharply increases g at the output layer.The output-layer increase corresponds to the reported downscaling effect of label smoothing on translation-model logits.

5 Conclusion

The paper presents three normalization-centric Transformer changes: PRENORM, SCALENORM, and FIXNORM. Together they improve low-resource NMT, while SCALENORM and FIXNORM remain comparable but faster in the high-resource setting; further optimizer and normalization combinations remain open for study.

  • 5 Conclusion: The paper proposes PRENORM, SCALENORM, and FIXNORM as simple normalization-centric changes for Transformer NMT.SCALENORM replaces LAYERNORM with scaled ℓ2 normalization using a single learned parameter, while FIXNORM fixes word-embedding norms.
  • 5 Conclusion: PRENORM is more stable and competent in low-resource NMT, whereas POSTNORM performs better for high-resource NMT in the original base Transformer regime.The conclusion contrasts the two normalization placements across resource settings.
  • 5 Conclusion: PRENORM + FIXNORM + SCALENORM significantly improves low-resource NMT, while FIXNORM and SCALENORM are comparably effective but faster in high-resource NMT.This conclusion states the combined low-resource benefit and the high-resource behavior of the latter two changes.
  • 5 Conclusion: Future work includes testing POSTNORM and PRENORM with RADAM, combining FIXNORM or SCALENORM with FIXUP, and exploring other ℓp norms.These directions are proposed rather than evaluated in the paper.

A Training details

The experiments use preprocessed multilingual translation pairs, standard Transformer dimensions and regularization, Adam optimization, and development-BLEU-based checkpoint selection. Evaluation reports tokenized BLEU with bootstrap significance testing and applies task-specific compound formatting for WMT English–German.

  • Data and preprocessing: The low-resource directions are en→he, en→vi, en→gl, en→sk, ar→en, gl→en, and sk→en as described across the selected corpora.The data is already preprocessed; experiments apply BPE with pair-dependent numbers of operations.
  • Model configuration: The model uses a 2048-dimensional feedforward sublayer and 512 dimensions elsewhere, matching the original Transformer configuration.Dropout is applied to sublayer outputs, ReLU, and attention weights; word dropout replaces tokens with UNK.
  • Training: Training uses Adam with batch size 4096, gradient clipping above global norm 1.0, early stopping, and validation-based learning-rate decay.Training stops at the epoch limit or when the learning rate reaches 10^-6, and the best checkpoint is selected by development BLEU.
  • Evaluation: Evaluation reports tokenized BLEU using multi-bleu.perl and measures statistical significance with bootstrap resampling.For WMT '14 English–German, compounds must be formatted in ATAT format before BLEU calculation for comparability.

B Further analysis

Further analysis examines whether SCALENORM’s low-resource gains reflect regularization or overall performance, using train/test perplexities. The reported evidence is suggestive but does not establish a conclusive trend.

  • B Further analysis: Train and test perplexities provide suggestive but inconclusive evidence about whether SCALENORM gains arise from improved regularization.The analysis compares smoothed train and test perplexities for PRENORM models using LAYERNORM and SCALENORM.
  • B Further analysis: For ar→en, gl→en, and sk→en, both train and test perplexities decrease slightly, with test perplexity decreasing more.For en→vi, train perplexity increases while test perplexity decreases by an equivalent amount.

C Listings

The listings define the FAIRSEQ baseline configuration and show how PRENORM, SCALENORM, and FIXNORM are enabled through implementation changes and command-line options.

  • FAIRSEQ: FAIRSEQ uses a POSTNORM Transformer base model with Adam, inverse-square-root scheduling, 4,000 warmup updates, dropout, label smoothing, and a 40-epoch limit.The configuration also uses a learning rate of 0.001, gradient clipping at 1.0, and a maximum of 8,192 tokens per batch.
  • PRENORM: PRENORM is enabled by adding encoder-normalize-before and decoder-normalize-before flags.
  • SCALENORM: SCALENORM replaces all LAYERNORM instances in the FAIRSEQ Transformer implementation.The listing identifies ScaleNorm as a module with a learnable scale parameter and epsilon value.
  • FIXNORM: FIXNORM initializes word embeddings uniformly in [−0.01, 0.01] and applies functional normalization.
  • Initialization: FAIRSEQ uses Xavier uniform initialization, which the authors describe as larger than their SMALLINIT; they conjecture large batches stabilize its training.
Loading 1910.05895v2…