PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

Djork-Arné Clevert, Thomas Unterthiner, Sepp Hochreiter

arXiv:1511.07289v5cs.LG

TL;DR

Deep networks need activation functions that reduce mean-shift effects while retaining stable, informative deactivation. The paper introduces exponential linear units (ELUs), whose negative saturation pushes mean activations toward zero and improves robustness, and reports significantly better activation-function performance across vision datasets, including a new CIFAR-100 state of the art.

  • Problem

    ReLUs alleviate vanishing gradients but produce positive mean activations, motivating activation functions that reduce bias-shift effects and speed learning.

  • Method

    ELUs combine an identity positive regime with negative values and a saturation plateau to push mean activations toward zero and provide noise-robust deactivation.

  • Results

    ELU networks significantly outperform other activation functions across vision datasets and achieve a best published CIFAR-100 test error of 24.28%.

  • Takeaways & Limitations

    ELUs provide a stable activation representation and improve learning and classification performance relative to the compared activation functions within the evaluated vision tasks.

  • Takeaways & Limitations

    On ImageNet, ELU networks are 5% slower than ReLU networks, requiring 12.15 hours versus 11.48 hours for 10,000 iterations.

Abstract

from arXiv · show

We introduce the "exponential linear unit" (ELU) which speeds up learning in deep neural networks and leads to higher classification accuracies. Like rectified linear units (ReLUs), leaky ReLUs (LReLUs) and parametrized ReLUs (PReLUs), ELUs alleviate the vanishing gradient problem via the identity for positive values. However, ELUs have improved learning characteristics compared to the units with other activation functions. In contrast to ReLUs, ELUs have negative values which allows them to push mean unit activations closer to zero like batch normalization but with lower computational complexity. Mean shifts toward zero speed up learning by bringing the normal gradient closer to the unit natural gradient because of a reduced bias shift effect. While LReLUs and PReLUs have negative values, too, they do not ensure a noise-robust deactivation state. ELUs saturate to a negative value with smaller inputs and thereby decrease the forward propagated variation and information. Therefore, ELUs code the degree of presence of particular phenomena in the input, while they do not quantitatively model the degree of their absence. In experiments, ELUs lead not only to faster learning, but also to significantly better generalization performance than ReLUs and LReLUs on networks with more than 5 layers. On CIFAR-100 ELUs networks significantly outperform ReLU networks with batch normalization while batch normalization does not improve ELU networks. ELU networks are among the top 10 reported CIFAR-10 results and yield the best published result on CIFAR-100, without resorting to multi-view evaluation or model averaging. On ImageNet, ELU networks considerably speed up learning compared to a ReLU network with the same architecture, obtaining less than 10% classification error for a single crop, single model network.

1 INTRODUCTION

The introduction motivates activation functions that reduce mean shifts toward zero to speed learning while preserving ReLU’s protection against vanishing gradients. It proposes ELUs, which combine negative outputs with saturation for a noise-robust deactivation state.

  • Motivation: ReLUs are popular because they produce sparse codes and alleviate the vanishing gradient problem through identity behavior for positive inputs.ReLUs output zero for non-positive arguments and act as the identity for positive arguments.
  • Motivation: Non-zero mean activations induce bias shifts in subsequent layers, while reducing those shifts brings standard gradients closer to natural gradients and speeds learning.The effect is larger when units are more correlated.
  • Motivation: Centering activations at zero has been proposed to speed learning by reducing Fisher-matrix off-diagonal terms and countering internal covariate shift.Batch normalization centers activations, while PRONG implicitly whitens them.
  • Proposed activation: LReLUs and PReLUs push mean activations toward zero through negative responses, but their linear negative regions do not ensure noise-robust deactivation.PReLUs learn the slope of the negative part and improve learning behavior.
  • Proposed activation: ELUs are proposed to combine negative outputs with saturation at smaller inputs, reducing deactivated-unit variation so precise deactivation arguments matter less.This design allows mean activations close to zero and can code the degree of presence of particular phenomena.

2 BIAS SHIFT CORRECTION SPEEDS UP LEARNING

The unit natural gradient mitigates bias shifts that cause oscillations and impede learning by correcting interactions with the bias unit. This correction shifts incoming mean activations toward zero and motivates activation functions with negative values.

  • Natural-gradient mechanism: The unit natural gradient corrects incoming-weight updates through the unit Fisher information matrix, restricting correction to interactions among weights leading to the unit.The unit Fisher matrix captures interactions of weights entering unit i, so the resulting natural gradient operates in that subspace.
  • Bias shift correction: Bias shifts change a unit’s mean activation, causing oscillations and impeding learning; the unit natural gradient mitigates or prevents them.The correction accounts for interactions between incoming units and the bias unit.
  • Mean correction: The bias-shift correction is equivalent to shifting incoming mean activations toward zero and scaling up the bias unit.Theorem 2 characterizes the mean correction as proportional to the q(z)-weighted incoming mean.
  • Design implication: Without the natural gradient, bias shifts can be reduced by centering incoming activations at zero or using activation functions with negative values.This motivates an activation function that combines negative values with the identity for positive inputs.

3 EXPONENTIAL LINEAR UNITS (ELUS)

ELUs use an identity-like positive branch and a saturating negative branch controlled by α, alleviating vanishing gradients while shifting mean activations toward zero. Their negative saturation reduces propagated variation and information, making representations more noise-robust.

  • ELU definition and gradient: ELUs use f(x)=x for x>0 and α(exp(x)−1) for x≤0, with α controlling negative-input saturation.The positive derivative is one, while α determines the negative saturation value.
  • ELU definition and gradient: ELUs alleviate vanishing gradients because their positive branch is the identity and has derivative one, unlike contractive tanh and sigmoid functions.ReLUs and LReLUs provide the same positive-branch mechanism for reducing vanishing gradients.
  • Mean activation shift: ELU negative values push mean activations closer to zero, bringing gradients closer to the natural gradient and enabling faster learning than ReLUs.This mean shift is the stated mechanism for faster learning.
  • Negative saturation: ELUs saturate at negative inputs, producing small derivatives that reduce forward-propagated variation and information and yield more noise-robust representations.The saturation behavior distinguishes ELUs from simply modeling increasingly negative activation values.

4 EXPERIMENTS USING ELUS

Experiments across MNIST, CIFAR-100, CIFAR-10, and ImageNet show that ELUs improve activation behavior, learning speed, accuracy, and generalization relative to alternative units and batch-normalized ReLU networks. ELUs achieved especially strong CIFAR-100 and ImageNet results, although they were modestly slower per iteration on ImageNet.

  • MNIST activation behavior: ELUs maintained smaller median activations than ReLUs and LReLUs throughout MNIST training, while reducing activation variation associated with bias-shift correction.The median activation varied much more in ReLU networks, indicating more persistent correction of bias shift from previous weight updates.
  • CIFAR-100 comparisons: 28.75(±0.24)% mean test error made ELUs more accurate than SReLUs, ReLUs, and LReLUs on CIFAR-100.SReLUs, ReLUs, and LReLUs achieved 29.35(±0.29)%, 31.56(±0.37)%, and 30.59(±0.29)%, respectively; ELUs also had lower training loss.
  • Generalization performance: ELU networks achieved a 6.55% CIFAR-10 test error, ranking second among the compared CNN architectures and placing them among the top 10 reported results.On CIFAR-100, ELU networks were reported as the best among the listed architectures.
  • ImageNet learning speed: 160k iterations were sufficient for ELU networks to reach 20% top-5 error on ImageNet, compared with 200k iterations for ReLU networks.A single-model, single-center-crop evaluation achieved a top-5 validation error below 10%.
  • ImageNet computational cost: 5% slower ImageNet training made ELUs modestly less efficient than ReLUs, requiring 12.15h versus 11.48h for 10k iterations.The paper attributes the small difference to activation functions’ minor influence on overall training time and anticipates faster exponential implementations.

5 CONCLUSION

ELUs are introduced as activation units for faster and more precise deep-network learning, using negative values to move mean activations toward zero. Their negative-regime saturation supports more robust and stable representations, while ImageNet results show earlier error reduction than ReLUs.

  • 5 CONCLUSION: ELUs enable faster and more precise learning in deep neural networks by using negative values to push mean activations closer to zero.This decreases the gap between the normal gradient and the unit natural gradient, thereby speeding learning.
  • 5 CONCLUSION: 160k iterations: ELU networks reached 20% top-5 error on ImageNet, whereas ReLU networks required 200k iterations for the same error rate.Both activation functions lead to convergence, but ELUs started reducing the error earlier.
  • 5 CONCLUSION: ELUs have a clear negative-regime saturation plateau that enables more robust and stable representations than LReLUs and PReLUs.The saturation plateau is presented as a distinguishing property from LReLUs and PReLUs.

A INVERSE OF BLOCK MATRICES

This section states the inverse formula for a positive definite block matrix and specializes it to a block containing a vector b and scalar c. It then reformulates the result using u.

  • A INVERSE OF BLOCK MATRICES: Lemma 1 gives the inverse of a positive definite matrix M written in block format with matrix A, vector b, and scalar c.The displayed inverse formula is introduced as the section’s main result.
  • A INVERSE OF BLOCK MATRICES: The proof invokes the general inverse formula for block matrices and identifies the matrices appearing on its right-hand side.The proof presents the block-matrix form before specializing it.
  • A INVERSE OF BLOCK MATRICES: Substituting B = b as a vector and C = c as a scalar yields the specialized expression, whose right-hand side includes matrices, vectors, and a scalar s.This is the stated specialization of the block inverse formula.
  • A INVERSE OF BLOCK MATRICES: The specialized inverse is additionally reformulated using u.The passage introduces this as an alternative formulation.

B QUADRATIC FORM OF MEAN AND INVERSE SECOND MOMENT

This section establishes quadratic-form identities for the mean and inverse second moment of a random variable. Using the Sherman–Morrison theorem and positive definiteness of the variance, it derives an upper bound and a related mixed-form expression.

  • Mean and inverse second moment: Lemma 2 gives a quadratic-form result for a random variable’s mean and inverse second moment.The displayed lemma statement is followed by formulas involving Var(a), E(a), and inverse second-moment terms.
  • Mean and inverse second moment: The Sherman–Morrison theorem is used to transform the relevant inverse-matrix expression.The derivation includes the form cT A−1b / (1 + bT A−1b).
  • Mean and inverse second moment: 1 is the upper bound obtained for the normalized quadratic form involving E(a) and Var−1(a).The inequality follows because Var(a) is positive definite.
  • Mean and inverse second moment: The mixed quadratic form is derived from Eq. (40) using Var(a) + E(a)ET(a).The section then develops the corresponding inverse-second-moment expression.
  • Mean and inverse second moment: The resulting identity rewrites 1 − ET_p(a)E−1(aaT)E(a) as 1 + (E(a) − Ep(a))T Var−1(a)E(a) divided by 1 + ET(a)Var−1(a)E(a).The equality is presented across the final displayed equations.

C VARIANCE OF MEAN ACTIVATIONS IN ELU AND RELU NETWORKS

In a five-hidden-layer MNIST network, the median activations of hidden units varied much more across epochs with ReLU than with ELU. The comparison was based on the variance of median-activation changes measured after each training epoch.

  • Variance comparison: ReLU networks showed substantially greater variance in median hidden-unit activation changes than ELU networks during MNIST training.The network had 5 hidden layers with 256 units each and was trained for 200 epochs at learning rate 0.01.
  • Variance comparison: Figure 7 displays the distributions of median-activation variances for units in each hidden layer.Median activations were calculated over the whole training set after every epoch.
Loading 1511.07289v5…