Source-linked AI summary
GLU Variants Improve Transformer
Noam Shazeer
TL;DR
Transformer FFN activations had limited evidence beyond ReLU, GELU, and related alternatives. The paper tests GLU-based replacements in T5 under matched parameter and computation budgets, finding better pre-training perplexities and stronger results on many downstream tasks. The authors report no apparent computational drawbacks but do not explain why the variants work.
Problem
The paper examines whether alternative GLU-based activations can improve Transformer feed-forward sublayers beyond commonly used ReLU or GELU activations.
Method
The authors replace the Transformer FFN’s first linear transformation and activation with GLU variants, reducing hidden size to match parameter and operation counts.
Results
The GLU variants seem to yield better denoising pre-training perplexities and better results on many downstream language-understanding tasks.
Takeaways & Limitations
GLU variants are simple to implement and have no apparent computational drawbacks in the reported transfer-learning setup.
Takeaways & Limitations
Comparisons reduce GLU-based layers’ hidden size because they use three weight matrices instead of two, constraining the comparison to matched parameter and computation budgets.
Abstract
from arXiv · showhide
Gated Linear Units (arXiv:1612.08083) consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Variations on GLU are possible, using different nonlinear (or even linear) functions in place of sigmoid. We test these variants in the feed-forward sublayers of the Transformer (arXiv:1706.03762) sequence-to-sequence model, and find that some of them yield quality improvements over the typically-used ReLU or GELU activations.
1 Introduction
The Transformer’s position-wise feed-forward network applies an activation between two learned linear transformations. This paper situates GLU variants as alternatives to the commonly used ReLU and related activations.
- 1 Introduction: The Transformer FFN maps each position’s hidden vector through two learned linear transformations with a ReLU between them.The standard formulation includes weight matrices W1 and W2 and bias vectors b1 and b2.
- 1 Introduction: A bias-free FFN variant is used following the T5 codebase.
- 1 Introduction: GELU and Swishβ are alternative nonlinear activations that have been proposed in place of ReLU.GELU is defined as xΦ(x), while Swishβ is defined as xσ(βx).
2 Gated Linear Units (GLU) and Variants
The paper replaces the Transformer FFN’s first linear transformation and activation with GLU-based variants. These variants use different gating functions while matching the original model’s parameter and computation budgets.
- 2 Gated Linear Units (GLU) and Variants: GLUs multiply two linear transformations component-wise, with one transformation passed through a sigmoid; omitting the activation yields a bilinear layer.
- 2 Gated Linear Units (GLU) and Variants: The paper defines additional variants by substituting ReLU, GELU, Swish, or no activation into the gated formulation.The named variants include ReGLU, GEGLU, SwiGLU, and Bilinear.
- 2 Gated Linear Units (GLU) and Variants: The proposed Transformer FFNs use GLU or a GLU variant instead of the first linear transformation and activation, with bias terms omitted.
- 2 Gated Linear Units (GLU) and Variants: Because GLU-based layers have three weight matrices instead of two, their hidden size is reduced by a factor of 2 3 to hold parameters and computation constant.
3 Experiments on Text-to-Text Transfer Transformer (T5)
The experiments evaluate GLU-based FFN variants in T5’s text-to-text transfer-learning setup under matched architecture and resource conditions. The variants achieve lower reported pre-training log-perplexities, while downstream results are noisy but favor them on most tasks.
- 3 Experiments on Text-to-Text Transfer Transformer (T5): The study tests FFN variants by pre-training an encoder-decoder Transformer on denoising and fine-tuning it on language-understanding tasks.
- 3 Experiments on Text-to-Text Transfer Transformer (T5): GLU-based models use dff = 2048 instead of the base model’s dff = 3072 to match parameter and operation counts.
- 3 Experiments on Text-to-Text Transfer Transformer (T5): 1.960 (0.005) is the reported heldout-set log-perplexity for FFNGEGLU, while FFNReGLU reports 1.944 (0.010).
- 3 Experiments on Text-to-Text Transfer Transformer (T5): 1.942 (0.004) is the reported heldout-set log-perplexity for FFNSwiGLU, compared with 1.982 (0.006) for FFNBilinear.
- 3 Experiments on Text-to-Text Transfer Transformer (T5): The new GLU variants perform best on most development-set tasks, although the fine-tuning results are noisy.
4 Conclusions
The paper concludes that GLU variants improve Transformer performance in its transfer-learning setting. It reports better pre-training perplexities and many stronger downstream results without apparent computational drawbacks, while leaving the mechanism unexplained.
- 4 Conclusions: The paper extends the GLU family and proposes using its variants in Transformer models.
- 4 Conclusions: The variants seem to produce better denoising-objective perplexities and better results on many downstream language-understanding tasks.
- 4 Conclusions: The proposed architectures are described as simple to implement and having no apparent computational drawbacks.
- 4 Conclusions: The paper offers no explanation for why the architectures seem to work.