PaperlayerView exact revision on arXiv

Source-linked AI summary

Review and Comparison of Commonly Used Activation Functions for Deep Neural Networks

Tomasz Szandała

arXiv:2010.09458v1cs.LGcs.NE

TL;DR

Choosing activation functions is difficult because neural networks face gradient and saturation challenges, so this paper compares common functions theoretically and practically. It finds ReLU and Leaky ReLU most successful on CIFAR-10, while concluding that no single function is universally optimal.

  • Problem

    Selecting activation functions is challenging because deep networks encounter zigzagging weights, vanishing gradients, saturation, and computational demands.

  • Method

    The paper compares common activation functions through theoretical descriptions, practical experiments, properties, advantages, disadvantages, and application contexts.

  • Results

    ReLU and Leaky ReLU achieved the highest CIFAR-10 accuracy, while ReLU trained over 5% faster than sigmoid and 15% faster than Leaky ReLU.

  • Takeaways & Limitations

    There is no ultimate activation-function choice; selection should follow the function’s properties, trade-offs, and intended application.

  • Takeaways & Limitations

    ReLU can exhibit the dying-ReLU problem under certain weight-distribution and gradient conditions.

Abstract

from arXiv · show

The primary neural networks decision-making units are activation functions. Moreover, they evaluate the output of networks neural node; thus, they are essential for the performance of the whole network. Hence, it is critical to choose the most appropriate activation function in neural networks calculation. Acharya et al. (2018) suggest that numerous recipes have been formulated over the years, though some of them are considered deprecated these days since they are unable to operate properly under some conditions. These functions have a variety of characteristics, which are deemed essential to successfully learning. Their monotonicity, individual derivatives, and finite of their range are some of these characteristics (Bach 2017). This research paper will evaluate the commonly used additive functions, such as swish, ReLU, Sigmoid, and so forth. This will be followed by their properties, own cons and pros, and particular formula application recommendations.

Introduction

Activation functions are critical tools in deep neural networks for mapping inputs to responses and enabling nonlinear computation, but training becomes difficult with multiple hidden layers because of issues such as vanishing gradients and saturation. This paper compares commonly used functions and their properties to guide activation-function selection for real-world applications.

  • Deep-Network Context: Increasing network depth has produced architectures ranging from twelve layers in AlexNet to one hundred and fifty-two layers in the largest ResNet architecture.The passage also identifies nineteen or sixteen layers in VGGNet variants and twenty-two layers in GoogleNet.
  • Role of Activation Functions: Activation functions are critical tools for mapping inputs to responses and representing nonlinear complex functions in artificial neural networks.They introduce non-linear properties into systems across different fields.
  • Training Challenges: Training deep neural networks becomes challenging because zigzagging weights, vanishing gradients, complicated formulas, and activation-function saturation can prolong learning.These challenges arise when networks contain multiple hidden layers.
  • Paper Contribution: The paper compares activation functions practically and theoretically to support appropriate choices and implementations in real-world applications.It summarizes current usage trends against state-of-the-art findings from practical deep-learning deployments.
  • Activation-Function Properties: A suitable activation function must be differentiable because backpropagation multiplies its derivatives, and it should support smooth weight updates.Smooth updates help avoid zigzagging during optimization, as illustrated by sigmoid-function behavior.

Step Function · Linear Activation Function

Step functions activate neurons only when inputs exceed a threshold, but their unrepresented derivative prevents effective gradient-based learning. Linear activation avoids thresholding and can support regression outputs, yet its constant derivative also limits gradient descent during training.

  • Step Function: Step functions activate a neuron when its input exceeds a specified threshold and otherwise leave it deactivated.The activation is therefore binary and condition-dependent.
  • Step Function: The step function is defined as f(x) = 0 for x ≤ T and f(x) = 1 for x > T.Its output changes according to the threshold condition.
  • Step Function: Primitive neural networks commonly use step functions with hidden layers, preferably a single layer, because the derivative’s learning value is not represented.The passage states that the derivative therefore has no effect in future learning.
  • Step Function: Zero gradients prevent step functions from updating during backpropagation, making progressive gradient or slope descent impossible.This is identified as the major problem with the step function.
  • Linear Activation Function: The linear activation function is expressed as f(x) = a * x, where a ∈ R represents proportional input activation.In the simplest case, the output can equal the input, while a may vary.
  • Linear Activation Function: Linear activation can activate many neurons simultaneously, and multiclass selection focuses on the neuron with the maximum output value.The passage presents this as an alternative to the step function.
  • Linear Activation Function: A constant derivative makes linear-function gradient descent constant and independent of x and the respective weights during backpropagation.The passage identifies this constant gradient as a training issue.
  • Linear Activation Function: Linear activation remains appropriate at the output layer of networks used for regression prediction.This is the stated setting where linear activation makes sense.

"S"-shaped Activation Functions · Hyperbolic Tangent Activation Function

S-shaped activations are nonlinear functions used to produce bounded outputs, but sigmoid has centrality and saturation limitations. Tanh addresses some of these issues through zero-centered outputs and larger near-zero derivatives, while still suffering vanishing gradients.

  • "S"-shaped Activation Functions: Sigmoid functions are nonlinear S-shaped activations with outputs ranging from 0 to 1.Their bounded range helps prevent activation blow-up compared with an unbounded linear function.
  • "S"-shaped Activation Functions: Sigmoid outputs change steeply when X lies between -2 and +2, supporting clearer prediction distinctions in shallow logic-function simulations.Small changes in X within this interval produce significant changes in Y.
  • "S"-shaped Activation Functions: Sigmoid saturation causes Y to respond only slightly to X changes near either end of the function.This limitation reduces responsiveness despite the function’s bounded activation range.
  • "S"-shaped Activation Functions: Sigmoid’s non-zero centrality always produces positive outputs, causing backpropagation error gradients for weighted inputs to share the same sign.The gradients are either both positive or both negative, restricting update directions.
  • Hyperbolic Tangent Activation Function: Tanh is introduced as an alternative when networks become stuck at edge values under sigmoid-only activation.It is also sigmoidal and is compared directly with sigmoid in the paper.
  • Hyperbolic Tangent Activation Function: Tanh maps outputs from -1 to 1, mapping negative inputs to negative outputs and inputs near zero to outputs near zero.These properties help prevent the network from becoming stuck at edge values.
  • Hyperbolic Tangent Activation Function: Tanh derivatives are noticeably larger than sigmoid derivatives near zero, helping minimize the cost function faster and reduce zigzagging.The passage identifies this derivative behavior as beneficial for finding local or global minima in big-data settings.
  • Hyperbolic Tangent Activation Function: Despite these advantages, tanh, like sigmoid, struggles with the vanishing-gradient problem.Sigmoid or tanh in a binary-classification final layer can produce quantities scaled from 0 to 1 for probability-based functions such as cross-entropy.

Softsign Activation Function

Softsign is a smoother alternative to tanh that grows polynomially rather than exponentially, supporting more effective learning by reducing vanishing-gradient and neuron-saturation problems, though it is more computationally expensive.

  • Softsign Activation Function: Softsign is smoother than the tanh activation function.
  • Softsign Activation Function: Softsign grows polynomially rather than exponentially, helping reduce vanishing-gradient difficulties and neuron saturation during learning.This gentler non-linearity was associated with better and faster learning and more effective learning.
  • Softsign Activation Function: Softsign is more expensive to compute than tanh because its derivatives are more complex.
  • Softsign Activation Function: Its gradient can sometimes yield extremely low or high values.

Vanishing Gradient Problem · Rectified Linear Unit Activation Functions · Basic Rectified Linear Unit (ReLU)

The section explains how vanishing gradients slow learning in deep networks and motivates wider-range or static derivatives. It then introduces ReLU as a piecewise-linear activation that supports efficient training and is particularly effective for deep classification networks.

  • Vanishing Gradient Problem: Sigmoid networks can exhibit vanishing gradients, with accuracy increasing slightly at two and three hidden layers before dropping at four layers.The example uses MNIST classification with 1–4 hidden layers, each containing 30 nodes.
  • Vanishing Gradient Problem: Backpropagation passes gradients through successive layers, allowing gradient magnitudes to indicate how quickly each hidden layer learns.The method represents gradients as derivatives of the cost function and compares the lengths of gradient vectors across layers.
  • Vanishing Gradient Problem: Deep neural networks can avoid vanishing gradients by using activation functions with static or wider-range derivatives.The section summarizes this as a remedy for the layerwise learning problem.
  • Rectified Linear Unit Activation Functions: Rectified linear unit activations transform weighted node inputs into outputs through piecewise-linear behavior, typically preserving positive inputs and outputting zero otherwise.These activations are presented as transformations from summed weighted inputs to strict or proportional outputs.
  • Basic Rectified Linear Unit (ReLU): Basic ReLU assigns zero to nonpositive values and returns the input itself for positive values, forming a ramp function.The section relates this behavior to half-wave rectification in electrical engineering.
  • Basic Rectified Linear Unit (ReLU): In 2011, ReLU was proven to enable more efficient training of deeper networks than commonly used sigmoid and hyperbolic tangent activations.The passage also identifies ReLU as the most popular deep-neural-network activation in research reported in 2018.
  • Basic Rectified Linear Unit (ReLU): ReLU’s function values and gradients are easy to calculate, enabling quick forward- and back-propagation steps.Its computational simplicity is presented as a practical advantage of the activation.
  • Basic Rectified Linear Unit (ReLU): For classification problems, especially those using multiple convolutional layers, ReLU and its minor variants are difficult to outperform.The passage contrasts ReLU with tanh, which is smoother and can approximate sine waves with fewer primitives.

Dying ReLU Problem · Dying ReLU - example

ReLU can suffer from the dying ReLU problem under certain weight and gradient conditions, causing a neuron to stop evolving. An XOR-network example demonstrates that this issue can prevent the network from achieving XOR behavior.

  • Dying ReLU Problem: Dying ReLU can occur when a large backpropagated gradient moves a ReLU node into a state where it no longer fires.The example begins with weights drawn from a low-variance Gaussian centered at +0.1, so most inputs initially activate the node; a large gradient can then move it adversely.
  • Dying ReLU Problem: The ReLU derivative is 0 for x ≤ 0 and 1 for x > 0.This piecewise derivative defines when the ReLU node passes gradients and when its gradient is zero.
  • Dying ReLU - example: The dying ReLU problem does not always appear, but simulations of a simple XOR-gate network show that it can occur.The reported simulations correspond to Fig. 8, described as a sample network fulfilling XOR-gate behavior.
  • Dying ReLU - example: The XOR example uses sigmoid for the output and ReLUs in the hidden-layer nodes to produce outputs of 0 or 1.Initial weights were set as potential randomly assigned values listed in Table 2.
  • Dying ReLU - example: The weights were adjusted to invoke dying ReLU, and the resulting pretrained XOR network did not produce perfect results.The authors therefore state that the weights need adjustment through backpropagation; the outputs are presented in Table 3.
  • Dying ReLU - example: Backpropagation updates weights by adding an adequate modifier, including updates between hidden and final neurons.The text introduces the update for w_31 in Equation 12 and then extends the procedure to all weights in Equation 13.
  • Dying ReLU - example: After training, neuron one is affected by dying ReLU, will never evolve, and prevents the network from achieving XOR-gate behavior.The conclusion is based on the post-training weights in Table 4, although the selected values are described as possible rather than impossible.

Leaky Rectified Linear Unit (ReLU)

Leaky ReLU introduces a small multiplier for negative inputs to reduce zero outputs and prevent dead nodes. Compared with standard ReLU, it can save energy in hardware implementations but offers no GPU difference.

  • Leaky Rectified Linear Unit (ReLU): Leaky ReLU applies a small multiplier α, such as 0.01, to values below 0 while retaining x for positive inputs.Its piecewise definition is f(x) = α*x for x ≤ 0 and x for x > 0.
  • Leaky Rectified Linear Unit (ReLU): The negative-part slope reduces the possibility of zero outputs, helping prevent ReLU neurons from becoming dead.The multiplier changes the inclination of the function’s negative portion.
  • Leaky Rectified Linear Unit (ReLU): Compared with standard ReLU, Leaky ReLU can save energy in hardware implementations such as minimized Internet of Things devices, but not on GPUs.Standard ReLU gives no activation value for negative values, whereas Leaky ReLU uses a nonzero negative slope.

Maxout · Softplus Activation Function

Maxout generalizes leaky ReLU by selecting the maximum output from two independent weight sets, while retaining ReLU-like benefits at substantially higher training cost. Softplus offers a smooth alternative to dead ReLU, with a sigmoid derivative useful for backpropagation.

  • Maxout: Maxout computes the maximum of two independent weight-set outputs, generalizing the leaky ReLU approach without arbitrarily choosing a negative-part parameter.The neuron selects the larger result from the two independently parameterized computations.
  • Maxout: Maxout doubles the number of parameters per neuron compared with ReLU neurons.This increased parameterization contributes directly to its greater training cost.
  • Maxout: Maxout retains ReLU benefits including nonsaturation and a linear operating regime.The passage describes these properties as benefits embraced by Maxout units.
  • Maxout: Although Maxout can provide the best results among ReLU-like functions, its training cost is highest, so it is recommended as a last resort.The passage notes successful use in phone speech as an example of this last-resort application.
  • Softplus Activation Function: Softplus is presented as an alternative to dead ReLU, with a smooth derivative that supports backpropagation.The passage identifies the smooth derivative as one of Softplus’s biggest advantages.
  • Softplus Activation Function: The derivative of Softplus equals the sigmoid function, and rectifier-trained networks can attain local minima of equal or greater quality despite a hard threshold at zero.The derivative–sigmoid relationship is stated twice in the supplied passage.

Swish Activation Function

Swish is presented as an alternative to ReLU that can address sigmoid’s vanishing-gradient problem and perform better than ReLU in experiments. Its main drawback is substantially higher computational cost during feed-forwarding and backpropagation.

  • Swish Activation Function: Introducing the non-zero parameter β produces a basic swish modification represented by f(x) = βxσ(βx).The modified function is described as using β as a parameter.
  • Swish Activation Function: As β approaches zero, the function becomes linear, whereas as β approaches one, it resembles ReLU.The simulation uses β as both a learned and a parameter.
  • Swish Activation Function: The Swish function is described as easily derivable, with its derivative given in Eq. 19.The derivative expression is provided in the cited passage.
  • Swish Activation Function: Swish is an alternative to ReLU that experiments report performing better than ReLU, described as highly efficient for deep learning.The passage also notes that ReLU remains important despite its limitations.
  • Swish Activation Function: Swish can deal with the vanishing-gradient problem that sigmoid cannot.This property is identified as a key advantage of the function.
  • Swish Activation Function: Swish has a much higher computational cost than ReLU for both feed-forwarding and backpropagation.The higher cost is also reported relative to its related activation-function variants.

Comparisons and Results Analysis

The comparison evaluates activation functions using training speed and classification accuracy in theoretical and real-world analysis. On CIFAR-10, ReLU and Leaky ReLU achieved the strongest accuracy, while ReLU led training and classification speed.

  • Evaluation criteria: The analysis compares activation-function properties, training speed, classification accuracy, equations, and range, using softmax only in final layers.All hidden nodes use the same activation function within each test series.
  • Experimental setup: The experiment uses CIFAR-10, comprising 60000 32x32 colour images in ten classes, with 25 training epochs and a simple two-convolution-layer network.The dataset includes 50000 training images and 10000 test images.
  • Accuracy results: ReLU and Leaky ReLU were the most successful accuracy functions, while all other networks completed the task with less than 70% accuracy.The results support ReLU’s reliability despite the potential dying-ReLU issue.
  • Training-time results: 5% less time: ReLU trained faster than sigmoid, and 15% less time: ReLU trained faster than Leaky ReLU.ReLU took over 5% less time than sigmoid and 15% less time than Leaky ReLU.
  • Classification-time results: 2.4 seconds: the ReLU network classified ten thousand images, compared with almost 2.5 seconds for Softplus.The passage presents this as empirical evidence favoring basic ReLU.

Conclusions

The paper concludes that no ultimate answer determines which activation function to choose, but its comprehensive review supports several theory-based recommendations. These recommendations address activation-function properties and application contexts in deep learning.

  • Conclusions: The smoother-near-zero alternative offers benefits comparable to ReLU but may slow networks because its formula is more complex.Its behavior is described as similar to ReLU while being smoother near zero.
  • Conclusions: The alternative matching leaky ReLU does not outperform ReLU according to the reviewed research, but may help in networks with dozens of layers.The recommendation is conditional on network depth.
  • Conclusions: One activation function is worth trying in very deep networks.The passage provides a depth-specific recommendation without identifying the function by name.
  • Conclusions: Another activation function is recommended for the output layer in classification networks.The passage specifies the application context but does not identify the function by name.
  • Conclusions: No ultimate answer determines which activation function should be chosen.The paper bases its recommendations on a comprehensive summary of activation functions used in deep learning.
Loading 2010.09458v1…