PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding and Improving Layer Normalization

Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, Junyang Lin

arXiv:1911.07013v1cs.LGcs.CLstat.ML

TL;DR

The paper addresses the unclear source of LayerNorm’s effectiveness, especially whether forward normalization explains its success. It analyzes backward-gradient derivatives and LayerNorm parameters, then proposes AdaNorm; AdaNorm outperforms LayerNorm on seven of eight datasets.

  • Problem

    The source of LayerNorm’s effectiveness remains unclear, and it is unknown whether forward-normalization theories explain its success.

  • Method

    The paper uses experiments and theoretical analysis to study LayerNorm’s backward gradients and parameters, then replaces bias and gain with AdaNorm’s adaptive transformation.

  • Results

    AdaNorm outperforms LayerNorm on seven datasets, while LayerNorm-simple outperforms LayerNorm on four datasets and achieves state-of-the-art En-Vi machine translation performance.

  • Takeaways & Limitations

    LayerNorm’s backward-gradient derivatives re-center and re-scale gradients, while its bias and gain increase over-fitting risk and often do not work.

  • Takeaways & Limitations

    AdaNorm requires its scaling function to be differentiable, maintain a constant average scaling weight, and keep the average output bounded; the analysis also requires |y_i| < 1/k.

Abstract

from arXiv · show

Layer normalization (LayerNorm) is a technique to normalize the distributions of intermediate layers. It enables smoother gradients, faster training, and better generalization accuracy. However, it is still unclear where the effectiveness stems from. In this paper, our main contribution is to take a step further in understanding LayerNorm. Many of previous studies believe that the success of LayerNorm comes from forward normalization. Unlike them, we find that the derivatives of the mean and variance are more important than forward normalization by re-centering and re-scaling backward gradients. Furthermore, we find that the parameters of LayerNorm, including the bias and gain, increase the risk of over-fitting and do not work in most cases. Experiments show that a simple version of LayerNorm (LayerNorm-simple) without the bias and gain outperforms LayerNorm on four datasets. It obtains the state-of-the-art performance on En-Vi machine translation. To address the over-fitting problem, we propose a new normalization method, Adaptive Normalization (AdaNorm), by replacing the bias and gain with a new transformation function. Experiments show that AdaNorm demonstrates better results than LayerNorm on seven out of eight datasets.

1 Introduction

The paper investigates why LayerNorm works and challenges the view that forward normalization alone explains its effectiveness. It identifies backward-gradient effects, questions LayerNorm’s bias and gain, and proposes AdaNorm as an adaptive alternative.

  • Background: LayerNorm was introduced to address BatchNorm’s limitations in RNNs and is used in Transformer models.It normalizes across neurons in a layer and is described as enabling faster Transformer training.
  • Research gap: The effectiveness of LayerNorm remains unclear because prior explanations emphasize forward distribution stability.The paper also asks whether theories developed for BatchNorm explain LayerNorm.
  • Findings: The derivatives of the mean and variance are more important than forward normalization for LayerNorm’s effectiveness.The authors report that these derivatives re-center and re-scale backward gradients.
  • Findings: LayerNorm-simple, which removes bias and gain, achieves better results than LayerNorm on four datasets and state-of-the-art performance on En-Vi machine translation.The paper attributes the poorer behavior of bias and gain to increased over-fitting risk.
  • Proposed method: AdaNorm replaces LayerNorm’s bias and gain with a transformation function that adaptively adjusts scaling weights from input values.It is evaluated across machine translation, language modeling, text classification, image classification, and dependency parsing.

2 Preliminaries

This section introduces LayerNorm’s input representation and parameters, then describes the experimental models, datasets, settings, and evaluation metrics used to study normalization across tasks.

  • LayerNorm algorithm: LayerNorm re-centers and re-scales an input vector x=(x_1, x_2, ..., x_H) using its mean and standard deviation, then applies gain g and bias b.The output is denoted h; gain and bias have the same dimension H as the input.
  • Experimental scope: The study evaluates normalization primarily with Transformer and Transformer-XL networks, while also using feed-forward neural networks to reduce architecture effects.The experiments cover machine translation, language modeling, text classification, image classification, and dependency parsing.
  • Machine translation: Machine translation experiments use En-De, De-En, and En-Vi datasets with PreNorm, Fairseq-based Transformer reimplementation, and BLEU evaluation.Dataset-specific configurations include BPE vocabularies, dropout, learning-rate, batch-size, optimizer, and warmup settings.
  • Language modeling: Language modeling uses the 100M-byte Enwiki83 corpus with a 12-layer Transformer-XL and average Bits-Per-Character evaluation.The model uses 512-dimensional layers, eight attention heads of dimension 64, dropout 0.1, and Adam with learning rate 0.00025.
  • Other tasks: Additional evaluations cover RT and SST5 text classification, MNIST image classification, and PTB dependency parsing with Transformer, convolutional, and MLP-based models.RT and SST5 use sentence classification, MNIST uses a three-layer convolutional network, and PTB follows a standard corpus split with an MLP parser.

3 Understanding LayerNorm

The analysis separates LayerNorm’s forward normalization from its backward-gradient effects and examines the roles of its affine parameters. It finds that gradient normalization, especially variance control in deeper networks, matters more than forward normalization, while removing bias and gain can improve results.

  • LayerNorm’s ablations separately evaluate bias and gain, forward normalization, and backward normalization.
  • 3.1 The Effect of the Bias and Gain in LayerNorm: LayerNorm-simple, which removes bias and gain, outperforms LayerNorm on four datasets and improves En-Vi by 0.4 BLEU and SST-5 by 1.31 ACC.It does not decrease performance on six datasets, and its En-Vi score of 31.6 is state of the art.
  • 3.1 The Effect of the Bias and Gain in LayerNorm: LayerNorm’s bias and gain can increase over-fitting: on En-Vi and Enwiki8, LayerNorm has lower training but higher validation loss or BPC than LayerNorm-simple.The paper attributes this risk to learned affine parameters that ignore testing-data input distributions.
  • 3.2 The Effect of Forward Normalization: DetachNorm preserves LayerNorm-simple’s forward re-centering and re-scaling while detaching mean and variance derivatives, enabling separate evaluation of backward effects.Detaching treats the mean and variance as constants during backward propagation; the implementation detaches standard deviation, the square root of variance.
  • 3.2 The Effect of Forward Normalization: DetachNorm performs worse than LayerNorm-simple on six datasets and even worse than the unnormalized baseline, indicating limited contribution from forward normalization and an important role for the derivatives.The gap between DetachNorm and LayerNorm-simple is used to assess the derivatives’ effect, while the gap between DetachNorm and “w/o Norm” assesses forward normalization.
  • 3.3 The Effect of the Derivatives of the Mean and Variance: Mean and variance derivatives re-center and re-scale backward gradients, collectively described as gradient normalization.The derivative of variance reduces the variance of the loss gradient, while the derivative of the mean re-centers it.
  • 3.3 The Effect of the Derivatives of the Mean and Variance: Detaching variance derivatives significantly harms deeper networks, showing that controlling gradient variance is necessary for those networks.The paper concludes that mean and variance derivatives are more important than forward normalization in LayerNorm.

4 AdaNorm

AdaNorm replaces LayerNorm’s fixed bias and gain with an adaptive transformation that controls scaling weights from inputs. It outperforms LayerNorm on seven datasets.

  • AdaNorm Algorithm: AdaNorm replaces LayerNorm’s bias and gain with a differentiable transformation that adaptively adjusts scaling weights from input values.The transformation is designed with fixed average scaling and bounded average output to support training stability.
  • AdaNorm Algorithm: AdaNorm’s transformation requires positive scaling weights, motivating the condition φ(y_i) = C(1 − ky_i) under bounded normalized inputs.The paper uses symmetry and a probability bound on normalized values to justify the constraint |y_i| < 1/k.
  • AdaNorm Algorithm: AdaNorm detaches the gradient of C(1 − ky) so the added term does not eliminate gradient re-centering and re-scaling.C is a hyper-parameter, ⊙ denotes elementwise multiplication, and k is recommended as 1/10.
  • Comparison between AdaNorm and LayerNorm: AdaNorm outperforms LayerNorm on seven datasets, including gains of 0.2 BLEU on En-Vi and 1.31 ACC on SST.Reported improvements also include En-De, De-En, RT, MNIST, and PTB.
  • Comparison between AdaNorm and LayerNorm: Compared with AdaNorm, LayerNorm has lower training loss but higher validation loss on En-Vi, PTB, and De-En.The paper associates lower validation loss with better convergence for AdaNorm.

5 Related Work

Normalization methods were developed to stabilize and improve deep-network training, with LayerNorm extending normalization to settings where BatchNorm has limitations. The mechanisms behind normalization effectiveness remain debated.

  • Normalization Methods: Normalization is presented as a technique that assists training by smoothing gradients, enabling larger learning rates, accelerating convergence, and improving generalization.BatchNorm was introduced to control layer-input distributions across mini-batches and reduce internal covariate shift.
  • Layer Normalization: LayerNorm extends normalization to recurrent and self-attention models by normalizing the mean and variance of inputs to neurons within a layer.Unlike BatchNorm, LayerNorm does not depend on mini-batch size.
  • Layer Normalization: LayerNorm has been applied in Transformer, BERT, and Transformer-XL frameworks.The passage describes these frameworks as state-of-the-art applications of LayerNorm.
  • Open Question: Studies of BatchNorm challenge the view that reduced internal covariate shift explains its success, instead relating effectiveness to smoother gradients.Whether this account explains LayerNorm remains unclear.

6 Conclusion

The paper analyzes LayerNorm’s mechanism, identifies backward-gradient effects as important, and reports that its learned bias and gain can increase over-fitting. It introduces AdaNorm as an alternative.

  • Conclusion: The paper finds that derivatives of the mean and variance help LayerNorm by re-centering and re-scaling backward gradients.This conclusion comes from experiments and theoretical analysis.
  • Conclusion: The paper reports that LayerNorm’s bias and gain increase over-fitting risk and do not work in most cases.These parameters are replaced in the proposed method rather than retained as fixed affine terms.
  • Conclusion: AdaNorm replaces LayerNorm’s bias and gain with an adaptive transformation that updates scaling weights from input values.The reported experiments show AdaNorm outperforming LayerNorm on seven datasets.

7 Appendix

The appendix-related materials describe evaluation datasets and model settings, then provide proof-oriented analyses of LayerNorm and AdaNorm properties.

  • Experimental Setup: The experiments cover translation, language modeling, sentiment classification, image recognition, and dependency parsing with task-specific datasets and models.The listed settings include Transformer, Transformer-XL, CNN, and MLP-based architectures.
  • Experimental Setup: The IWSLT English-Vietnamese setup contains 133K training sentence pairs, with TED tst2012 for validation and TED tst2013 for testing.BPE is used for input and output vocabularies.
  • Gradient Analysis: The proof analysis uses vectors orthogonal to the all-ones vector and normalized inputs to characterize backward-gradient behavior.It expands relevant vectors into orthogonal bases and analyzes coefficient constraints.
  • Gradient Analysis: The appendix includes analyses of standard LayerNorm gradients and variants that detach gradients of the mean and variance.These calculations support the paper’s backward-gradient analysis.
Loading 1911.07013v1…