PaperlayerView exact revision on arXiv

Source-linked AI summary

GLU Variants Improve Transformer

Noam Shazeer

arXiv:2002.05202v1cs.LGcs.NEstat.ML

TL;DR

Transformer FFN activations had limited evidence beyond ReLU, GELU, and related alternatives. The paper tests GLU-based replacements in T5 under matched parameter and computation budgets, finding better pre-training perplexities and stronger results on many downstream tasks. The authors report no apparent computational drawbacks but do not explain why the variants work.

  • Problem

    The paper examines whether alternative GLU-based activations can improve Transformer feed-forward sublayers beyond commonly used ReLU or GELU activations.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The authors replace the Transformer FFN’s first linear transformation and activation with GLU variants, reducing hidden size to match parameter and operation counts.

    Source: §2 · p. 2

  • Results

    The GLU variants seem to yield better denoising pre-training perplexities and better results on many downstream language-understanding tasks.

    Source: §4 · pp. 3–4, §3.3 · p. 3

  • Takeaways & Limitations

    GLU variants are simple to implement and have no apparent computational drawbacks in the reported transfer-learning setup.

    Source: §4 · p. 3

    Comparisons reduce GLU-based layers’ hidden size because they use three weight matrices instead of two, constraining the comparison to matched parameter and computation budgets.

    Source: §2 · p. 2

Abstract

from arXiv · show

Gated Linear Units (arXiv:1612.08083) consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Variations on GLU are possible, using different nonlinear (or even linear) functions in place of sigmoid. We test these variants in the feed-forward sublayers of the Transformer (arXiv:1706.03762) sequence-to-sequence model, and find that some of them yield quality improvements over the typically-used ReLU or GELU activations.

1 Introduction

The Transformer’s position-wise feed-forward network applies an activation between two learned linear transformations. This paper situates GLU variants as alternatives to the commonly used ReLU and related activations.

1 Introduction

  • The Transformer FFN maps each position’s hidden vector through two learned linear transformations with a ReLU between them.The standard formulation includes weight matrices W1 and W2 and bias vectors b1 and b2.Source: §1 · p. 1
  • A bias-free FFN variant is used following the T5 codebase.Source: §1 · p. 1
  • GELU and Swishβ are alternative nonlinear activations that have been proposed in place of ReLU.GELU is defined as xΦ(x), while Swishβ is defined as xσ(βx).Source: §1 · p. 1

2 Gated Linear Units (GLU) and Variants

The paper replaces the Transformer FFN’s first linear transformation and activation with GLU-based variants. These variants use different gating functions while matching the original model’s parameter and computation budgets.

2 Gated Linear Units (GLU) and Variants

  • GLUs multiply two linear transformations component-wise, with one transformation passed through a sigmoid; omitting the activation yields a bilinear layer.Source: §2 · p. 1
  • The paper defines additional variants by substituting ReLU, GELU, Swish, or no activation into the gated formulation.The named variants include ReGLU, GEGLU, SwiGLU, and Bilinear.Source: §2 · pp. 1–2
  • The proposed Transformer FFNs use GLU or a GLU variant instead of the first linear transformation and activation, with bias terms omitted.Source: §2 · p. 2
  • Because GLU-based layers have three weight matrices instead of two, their hidden size is reduced by a factor of 2 3 to hold parameters and computation constant.Source: §2 · p. 2

3 Experiments on Text-to-Text Transfer Transformer (T5)

The experiments evaluate GLU-based FFN variants in T5’s text-to-text transfer-learning setup under matched architecture and resource conditions. The variants achieve lower reported pre-training log-perplexities, while downstream results are noisy but favor them on most tasks.

3 Experiments on Text-to-Text Transfer Transformer (T5)

  • The study tests FFN variants by pre-training an encoder-decoder Transformer on denoising and fine-tuning it on language-understanding tasks.Source: §3 · p. 2
  • GLU-based models use dff = 2048 instead of the base model’s dff = 3072 to match parameter and operation counts.Source: §3.1 · p. 2, Table 1 · p. 2
  • 1.960 (0.005) is the reported heldout-set log-perplexity for FFNGEGLU, while FFNReGLU reports 1.944 (0.010).Source: §3.1 · p. 2
  • 1.942 (0.004) is the reported heldout-set log-perplexity for FFNSwiGLU, compared with 1.982 (0.006) for FFNBilinear.Source: §3.1 · p. 2
  • The new GLU variants perform best on most development-set tasks, although the fine-tuning results are noisy.Source: §3.3 · p. 3

4 Conclusions

The paper concludes that GLU variants improve Transformer performance in its transfer-learning setting. It reports better pre-training perplexities and many stronger downstream results without apparent computational drawbacks, while leaving the mechanism unexplained.

4 Conclusions

  • The paper extends the GLU family and proposes using its variants in Transformer models.Source: §4 · p. 3
  • The variants seem to produce better denoising-objective perplexities and better results on many downstream language-understanding tasks.Source: §4 · pp. 3–4
  • The proposed architectures are described as simple to implement and having no apparent computational drawbacks.Source: §4 · p. 3
  • The paper offers no explanation for why the architectures seem to work.Source: §4 · p. 3

Related papers

Loading 2002.05202v1…