PaperlayerView exact revision on arXiv

Source-linked AI summary

Gaussian Error Linear Units (GELUs)

Dan Hendrycks, Kevin Gimpel

arXiv:1606.08415v5cs.LG

TL;DR

Existing activations such as ReLU and ELU offer limited alternatives for modeling neuron outputs. The paper introduces GELU, which weights inputs by their value, and finds it matches or exceeds ReLU and ELU across computer vision, NLP, and speech tasks.

  • Problem

    ReLU and ELU provide limited alternatives for nonlinear activation design, motivating evaluation of another activation across diverse tasks.

  • Method

    The paper introduces GELU, defined as xΦ(x), a deterministic expected transformation that weights inputs by their values.

  • Results

    GELU matches or exceeds ReLU and ELU across computer vision, natural language processing, and automatic speech recognition tasks.

  • Takeaways & Limitations

    GELU is a viable alternative to ReLU and ELU across the numerous evaluated datasets.

Abstract

from arXiv · show

We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function. The GELU activation function is $xΦ(x)$, where $Φ(x)$ the standard Gaussian cumulative distribution function. The GELU nonlinearity weights inputs by their value, rather than gates inputs by their sign as in ReLUs ($x\mathbf{1}_{x>0}$). We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and find performance improvements across all considered computer vision, natural language processing, and speech tasks.

1 INTRODUCTION

The introduction frames GELU as a new, probabilistically motivated nonlinearity that addresses limitations of sigmoid and ReLU activations. It reports that GELU matches or exceeds ReLU and ELU models across vision, language, and speech tasks.

  • Activation-function background: Sigmoid activations enable backpropagation and a firing-rate interpretation, but deeper networks trained less effectively with them than with ReLU.ReLU makes hard gating decisions based on an input’s sign.
  • GELU contribution: The Gaussian Error Linear Unit (GELU) is introduced as a new nonlinearity related to Adaptive Dropout through an expectation over a modification.This relationship suggests a more probabilistic view of a neuron’s output.
  • Empirical scope: GELU matches or exceeds ReLU and ELU models across computer vision, natural language processing, and automatic speech recognition tasks.The comparison spans three application areas: computer vision, natural language processing, and automatic speech recognition.

2 GELU FORMULATION

GELU combines input-dependent stochastic masking with a deterministic expected transformation, yielding xΦ(x) under the standard Gaussian CDF. The formulation is motivated by ReLU, dropout, and zoneout, while fixing the Gaussian parameters at μ = 0 and σ = 1 without introducing new hyperparameters.

  • Motivation: GELU merges properties of ReLU, dropout, and zoneout by multiplying inputs by a stochastic zero-one mask.ReLU applies a deterministic zero-one multiplication, dropout stochastically multiplies by zero, and zoneout stochastically multiplies by one.
  • Input-dependent masking: The underlying mask preserves non-determinism while depending on input magnitude, dropping inputs more often as x decreases.This input-dependent zero-or-identity transformation is compared with adaptive dropout, which uses a logistic rather than standard normal distribution.
  • Gaussian choice: The standard normal distribution is chosen because neuron inputs tend to follow a normal distribution, especially with Batch Normalization.GELU therefore uses the standard Gaussian cumulative distribution function Φ(x) for its input-dependent weighting.
  • GELU formulation: GELU is defined as the expected stochastic transformation Φ(x) × x + (1 − Φ(x)) × 0 = xΦ(x).The stochastic mask selects a zero or identity transformation, and GELU replaces that randomness with its expectation.
  • Design choices: Alternative CDFs yield related nonlinearities such as the Sigmoid Linear Unit xσ(x), but GELU fixes μ = 0 and σ = 1 throughout the experiments.The Gaussian mean and standard deviation could instead be learnable hyperparameters.

3 GELU EXPERIMENTS

The experiments compare GELU with ELU and ReLU across vision, language, and speech tasks using varied architectures and training settings. Reported results include robustness matching or exceeding the baselines on noisy MNIST and slightly lower TIMIT test error for GELU.

  • Experimental scope: GELU, ELU, and ReLU are evaluated on MNIST classification and autoencoding, Tweet part-of-speech tagging, TIMIT frame recognition, and CIFAR-10/100 classification.The datasets span computer vision, natural language processing, and speech, with the listed training, validation, and test sizes.
  • Experimental scope: The replicated fully connected comparison uses 8-layer, 128-neuron-wide networks trained for 50 epochs with batch size 128.GELUs use µ = 0 and σ = 1, ReLUs are compared with ELUs using α = 1.
  • MNIST robustness: GELU robustness matches or exceeds ELU and ReLU as uniform input noise increases on MNIST classifiers trained without dropout.Robustness is measured through test accuracy decline and log-loss increase across noise strengths up to a = 3.
  • Training procedure: The comparison uses Adam, unit-norm row initialization, dropout evaluation, and learning-rate tuning over 10^-3, 10^-4, and 10^-5.The learning rates are selected using 5k validation examples from the training set, with median results reported in the supplied passage.
  • TIMIT frame classification: 29.3% is the median TIMIT test error for GELU, compared with 29.5% for ReLU and 29.6% for ELU.The five-layer, 2048-neuron-wide classifier uses 0.5 dropout, Adam, and median test error selected at the lowest validation error.
  • CIFAR classification: GELU is also evaluated in shallow CIFAR-10 and deep CIFAR-100 convolutional networks, including a 40-layer wide residual network with widening factor 4.The CIFAR-10 model uses batch normalization without data augmentation, while the CIFAR-100 residual network uses Nesterov momentum and dropout keep probability 0.7.

4 DISCUSSION

The discussion positions GELU as a smooth alternative to ReLU with distinct curvature and non-monotonicity, while offering practical guidance for optimization and Gaussian-CDF approximation. It reports that GELU outperformed previous nonlinearities across several experiments.

  • Relationship to ReLU: GELU can be viewed as a smooth approximation to ReLU, becoming ReLU as σ → 0 when μ = 0 and matching it asymptotically.For μ = 0 and σ = 1, GELU is xΦ(x), replacing ReLU’s binary indicator with a smooth Gaussian CDF.
  • Empirical comparison: Across several experiments, GELU outperformed previous nonlinearities while retaining some similarities to ReLU and ELU.The discussion frames GELU as related to, but notably different from, those activations.
  • Distinctive properties: Unlike convex, monotonic ReLUs and ELUs, GELU is non-convex, non-monotonic, nonlinear for positive inputs, and curved at all points.The discussion suggests these properties may help GELUs approximate complicated functions more easily.
  • Practical guidance: The authors advise using momentum during GELU training and emphasize that a close Gaussian-CDF approximation is important.They report that SiLU, xσ(x), performs worse than GELUs.
  • Practical guidance: The paper uses xΦ(x) ≈ xσ(1.702x) or xΦ(x) ≈ 0.5x[1 + tanh(√(2/π)(x + 0.044715x^3))] as fast, easy-to-implement approximations.The latter approximation was used in every experiment in the paper.

5 CONCLUSION

Across the numerous datasets evaluated, GELU consistently exceeded the accuracy of ELU and ReLU, establishing it as a viable alternative to previous nonlinearities.

  • GELU consistently exceeded ELU and ReLU accuracy across the numerous datasets evaluated, making it a viable alternative to previous nonlinearities.

A NEURAL NETWORK ARCHITECTURE FOR CIFAR-10 EXPERIMENTS

This section presents the neural network architecture used for CIFAR-10 experiments.

  • The paper provides a neural network architecture for CIFAR-10.
  • The architecture is presented in Table 1.
  • The table concerns CIFAR-10 experiments.

B HISTORY OF THE GELU AND SILU

The paper introduced and coined the Sigmoid Linear Unit (SiLU) as x · σ(x), while later work proposed the same activation under other names. GELU subsequently became the default activation for state-of-the-art Transformers and more commonly used than SiLU.

  • Origin of SiLU: The paper introduced and coined the Sigmoid Linear Unit (SiLU) as x · σ(x) in 2016.The work discussed smoother ReLU activation functions and their relation to stochastic regularizers before being submitted to ICLR and publicly released.
  • Rediscovery and naming: In 2017, Elfwing et al. proposed the same activation, x · σ(x), calling it “SIL,” while Google Brain later proposed it without citing either prior work.Elfwing subsequently updated the name to “SiLU” after contact from the authors.
  • Rediscovery and naming: Google Brain continued calling the activation “swish” despite acknowledging that prior works had proposed the same function.The first author publicly attributed the omission to an insufficient literature search, and the paper was updated a week later.
  • Rediscovery and naming: Nearly all users adopted the original “swish” function without β, with β = 1 making it identical to SiLU.TensorFlow’s implementation was called “swish,” and its default setting removed β.
  • GELU adoption: GELU became the default activation for state-of-the-art Transformers through its use in BERT and GPT, and is now substantially more commonly used than SiLU.This marked a shift in practical adoption from SiLU toward GELU.
Loading 1606.08415v5…