PaperlayerView exact revision on arXiv

Source-linked AI summary

Activation Functions: Comparison of trends in Practice and Research for Deep Learning

Chigozie Nwankpa, Winifred Ijomah, Anthony Gachagan, Stephen Marshall

arXiv:1811.03378v1cs.LGcs.CV

TL;DR

Deep neural networks require suitable activation functions, yet selecting them across architectures and application domains remains difficult, while practical usage trends are not well compiled against research results. This paper surveys existing activation functions, summarizes their applications and reported advantages, and compares practical deployment trends with state-of-the-art research. It finds that newer state-of-the-art activation functions are rarely used in practice, which makes the compilation relevant for activation-function selection.

  • Problem

    Selecting appropriate activation functions across deep-learning architectures and application domains remains challenging, while practical trends have not been comprehensively compared with research results.

  • Method

    The paper surveys and compiles activation functions used in deep learning, their reported advantages, application trends, and practical usage.

  • Results

    Newly developed state-of-the-art activation functions are rarely used in practice, which remains dependent on tested and proven functions.

  • Takeaways & Limitations

    The compilation supports decisions about activation-function choices for deep-learning applications and deployment.

Abstract

from arXiv · show

Deep neural networks have been successfully used in diverse emerging domains to solve real world complex problems with may more deep learning(DL) architectures, being developed to date. To achieve these state-of-the-art performances, the DL architectures use activation functions (AFs), to perform diverse computations between the hidden layers and the output layers of any given DL architecture. This paper presents a survey on the existing AFs used in deep learning applications and highlights the recent trends in the use of the activation functions for deep learning applications. The novelty of this paper is that it compiles majority of the AFs used in DL and outlines the current trends in the applications and usage of these functions in practical deep learning deployments against the state-of-the-art research results. This compilation will aid in making effective decisions in the choice of the most suitable and appropriate activation function for any given application, ready for deployment. This paper is timely because most research papers on AF highlights similar works and results while this paper will be the first, to compile the trends in AF applications in practice against the research results from literature, found in deep learning research to date.

I. INTRODUCTION

Deep learning extends machine learning through multilayer representation learning, but increasing depth makes network behavior and training more difficult to understand and optimize. This survey focuses on activation functions as an important choice for learning, generalization, and deployment across applications.

  • Deep learning foundations: Deep learning uses multiple transformations to convert raw inputs into increasingly abstract representations.These representations are learned by detecting relationships and patterns in training data.
  • Deep learning foundations: As architectures grew from five layers in LeNet5 to over 1,200 layers in Stochastic Depth networks, understanding hidden-layer operations became increasingly necessary.The cited architectures include AlexNet, VGGNet, GoogleNet, and ResNet as intermediate depth milestones.
  • Challenges: Deep neural networks are difficult to develop because they must produce learned complex functions effectively.Training-related issues have been a continuing research focus.
  • Challenges: Gradient flow can become sharp, slow, or zero in different directions, complicating the selection of learning parameters.Vanishing and exploding gradients are identified as major activation-function issues.
  • Activation-function focus: Activation functions are studied because they affect learning and generalization alongside other neural-network hyperparameters.The paper contrasts activation-function use across domains including classification, segmentation, translation, cancer detection, and forecasting.
  • Paper scope: The paper compiles activation-function application trends and compares practical usage with state-of-the-art research results.It is organized into six sections covering deep learning, activation functions, summaries, application trends, discussion, and conclusions.

II. ACTIVATION FUNCTIONS

Activation functions transform neural-network outputs and determine how information propagates through hidden and output layers. They introduce nonlinearity, support gradient-based learning, and control outputs across many deep-learning applications.

  • Role and placement: Activation functions transform weighted inputs and biases into outputs used by neural-network layers.Their position depends on their network role, with hidden-layer functions supporting propagation and output-layer functions controlling predictions.
  • Role and placement: Nonlinear activation outputs are required because linear mappings alone cannot learn higher-order patterns in deeper networks.The paper states that differentiability is needed for backpropagation.
  • Applications: Deep-learning systems use activation functions across image, speech, segmentation, translation, medical, forecasting, and autonomous-driving applications.The output layer commonly uses activation functions for classification or prediction probabilities.
  • Gradient behavior: Activation functions help regulate gradient values within limits as repeated derivative multiplication can cause vanishing or exploding gradients.Values below 1 can shrink toward zero, while values above 1 can grow toward infinity.
  • Survey scope: The paper compiles existing activation functions together with advantages reported for the respective functions in the literature.This compilation covers functions used in deep-learning applications and their reported benefits.

III. SUMMARY OF ACTIVATION FUNCTIONS

The paper organizes activation functions by presenting main functions first and then their variants or improvements. This summary is arranged by function families rather than chronologically.

  • Organization: The activation-function summary reviews functions used in deep architectures and their evolution across applications.The section frames activation-function research and application as an ongoing core research field.
  • Organization: Functions are ordered with the main activation functions first, followed by their improved variants.The authors explicitly state that the arrangement is not chronological.

A. Sigmoid Function

The sigmoid family includes bounded differentiable functions used especially for output probabilities and binary classification, but standard sigmoid can hinder deep-network training. Its variants address selected computational or optimization needs.

  • Sigmoid Function: The Sigmoid AF is a bounded, differentiable nonlinear function used mostly in feedforward neural networks.It is described as smooth with positive derivatives for real inputs.
  • Sigmoid Function: Sigmoid functions appear in output layers for probability-based prediction and have been applied successfully to binary classification and logistic regression.The paper also notes their use in other neural-network domains.
  • Sigmoid Function: Standard Sigmoid suffers from damped gradients, saturation, slow convergence, and non-zero-centred outputs during deep backpropagation.These drawbacks motivated proposals such as the hyperbolic tangent function.
  • Sigmoid variants: Hard sigmoid offers lower computational cost than soft sigmoid in specialized hardware or software implementations.The authors report promising results for deep-learning binary-classification tasks.
  • Sigmoid variants: SiLU is a reinforcement-learning-based approximation function computed as sigmoid multiplied by its input.The paper restricts its use to hidden layers of reinforcement-learning systems and reports that it outperformed ReLU in the cited response.

3) Derivative of Sigmoid-Weighted Linear Units (dSiLU):

The dSiLU gradient supports gradient-descent updates and has an overshooting Sigmoid-like response; the paper reports that dSiLU significantly outperformed standard Sigmoid. The surrounding comparison situates activation functions by response shape and application.

  • dSiLU is used for gradient-descent learning updates of neural-network weight parameters.
  • The dSiLU response resembles an overshooting Sigmoid function.
  • dSiLU significantly outperformed the standard Sigmoid function.
  • B. Hyperbolic Tangent Function (Tanh): Tanh is a smoother zero-centred function with outputs ranging from -1 to 1.
  • B. Hyperbolic Tangent Function (Tanh): Tanh provided better training performance than Sigmoid for multi-layer neural networks but did not solve Sigmoid's vanishing-gradient problem.
  • Softmax produces class probabilities between 0 and 1 whose values sum to 1, making it suitable for multi-class models.

D. Softsign

Softsign is a nonlinear activation function used mainly in regression and also in speech-related deep-learning tests. The section contrasts its polynomial convergence with tanh's exponential convergence and reviews ReLU's practical trade-offs.

  • D. Softsign: Softsign is a nonlinear activation function used mainly in regression and also applied to speech systems.Reported speech-system results were described as promising.
  • D. Softsign: Softsign converges in polynomial form, unlike tanh, which converges exponentially.
  • ReLU accelerates computation by avoiding exponentials and divisions and introduces sparsity in hidden units.
  • ReLU can overfit more easily than Sigmoid, although dropout was adopted to reduce this effect.
  • ReLU training can cause gradients and neurons to die, preventing later weight updates and hindering learning.
  • Leaky ReLU maintains nonzero gradients across training, but showed no significant improvement over standard ReLU except in sparsity and dispersion.

2) Parametric Rectified Linear Units (PReLU):

PReLU extends ReLU by learning its negative slope during backpropagation, while related ReLU variants modify negative-region behavior or learnable piecewise parameters. Reported studies found improvements over standard ReLU in several classification settings.

  • 2) Parametric Rectified Linear Units (PReLU): PReLU adaptively learns the negative part of its activation while retaining a linear positive part.Its negative-slope parameter is learned through backpropagation.
  • 2) Parametric Rectified Linear Units (PReLU): Setting the PReLU parameter a_i to 0 recovers the standard ReLU.
  • 2) Parametric Rectified Linear Units (PReLU): PReLU performed better than ReLU in large-scale image recognition.
  • LReLU, RReLU, and PReLU were reported to perform better than ReLU on classification tasks.
  • SReLU uses three piecewise-linear functions with four learnable parameters and showed improved results on CIFAR-10, ImageNet, and MNIST.

F. Softplus Function

Softplus is a smooth ReLU variant with nonzero gradients, while ELU and PELU modify negative-region behavior to address optimization and centering concerns. Reported comparisons associate these functions with faster convergence, improved generalization, or reduced bias shifts in specific settings.

  • F. Softplus Function: Softplus is a smooth ReLU variant with smoothing and nonzero-gradient properties.The paper associates these properties with stabilization and performance in networks using Softplus units.
  • F. Softplus Function: Softplus achieved improved performance with fewer epochs to convergence than ReLU and Sigmoid in a reported comparison.
  • ELU's negative values push mean unit activation closer to zero, reducing bias shifts and computational complexity while improving learning speed.
  • ELU provides a negative saturation plateau and was reported to learn faster and generalize better than ReLU and LReLU in specific network structures above five layers.
  • ELU does not centre values at zero, motivating parametric ELU variants that address this issue.
  • Setting a, b, and c to 1 in PELU recovers the original ELU activation function.

2) Scaled Exponential Linear Units (SELU):

SELU is presented as a self-normalising activation whose propagated outputs converge toward zero mean and unit variance. The section also surveys Maxout and Swish, including their reported applications, properties, and limitations.

  • Scaled Exponential Linear Units (SELU): SELU induces self-normalising properties, with mean and variance converging toward zero mean and unit variance across multiple layers during training.The paper describes this behaviour as making SELU suitable for deep learning applications.
  • Scaled Exponential Linear Units (SELU): SELU is reported to avoid vanishing and exploding gradient problems and, with strong regularisation, learn robust features efficiently.The paper contrasts these mappings with those derived using ReLU, scaled ReLU, sigmoid, LReLU, and tanh.
  • Scaled Exponential Linear Units (SELU): SELU has been applied successfully to classification tasks and deep genetic mutation computation.
  • Maxout: Maxout generalises ReLU and leaky ReLU without dying neurons or saturation, and has been tested successfully in phone recognition.
  • Maxout: Maxout doubles the parameters used in all neurons, making the function computationally expensive.
  • Swish: Swish combines sigmoid with the input, producing a smooth, non-monotonic function that is bounded below and unbounded above.The paper gives Swish as f(x) = x · sigmoid(x) = x / (1 + e^-x).
  • Swish: Swish is reported to improve optimisation and generalisation, avoid vanishing gradients, and outperform ReLU on deep learning classification tasks.The paper attributes improved information propagation during training to Swish.

J. ELiSH

ELiSH combines ELU- and sigmoid-related behavior, with sigmoid improving information flow and linear components addressing vanishing gradients. The function has been successfully applied to ImageNet across different deep convolutional architectures, while HardELiSH is presented as its hard variant and tested on ImageNet classification.

  • ELiSH: ELiSH combines ELU and Sigmoid functions, sharing common properties with Swish.
  • ELiSH properties: The sigmoid component improves information flow, while the linear parts eliminate vanishing-gradient issues.
  • Applications: ELiSH has been successfully applied to ImageNet using different deep convolutional architectures.
  • HardELiSH: HardELiSH is the hard variant of ELiSH, combining HardSigmoid with ELU in the negative region and Linear with HardSigmoid in the positive region.
  • HardELiSH evaluation: HardELiSH was tested on the ImageNet classification dataset.

V. DISCUSSIONS

The survey catalogs activation-function variants and contrasts research findings with their use in practical deep-learning architectures. It finds that newer functions often outperform established ones in studies, while practice continues to favor proven functions such as ReLU.

  • Activation-function landscape: The survey identifies multiple activation-function families and variants, including ReLU, LReLU, PReLU, RReLU, SReLU, ELU, PELU, and SELU.It also summarizes activation functions and their computation formulae for deep-learning applications.
  • Activation-function landscape: ReLU has four variants, while Sigmoid has three variants; other summarized functions include ELU variants PELU and SELU.The listed Sigmoid variants include HardSigmoid, SiLU, and dSiLU, while ReLU variants include LReLU, PReLU, RReLU, and SReLU.
  • Research findings: LReLU, RReLU, and PReLU were validated as performing better than ReLU on classification tasks, although some variants lack theoretical justification for their results.The survey also reports ELU as faster-learning than ReLU, with this assertion validated on MNIST recognition.
  • Research findings: Parametric activation functions use learnable parameters from the dataset, a development observed in SReLU, PELU, and PReLU.PELU and PReLU are described as parametric functions developed from exponential and linear units, respectively.
  • Application patterns: Most activation functions were developed and tested with convolutional neural networks on classification datasets, indicating that these settings dominate the surveyed applications.ReLU and SiLU were developed on restricted Boltzmann machines, while LReLU was developed with neural-network acoustic models.
  • Practice versus research: Although newer activation functions appear to outperform older functions such as ReLU, current architectures still rely on tested functions, and newer functions are rarely used in practice.The survey notes that SeNet uses ReLU in hidden layers and Sigmoid at the output.

VI. CONCLUSION

The paper surveys activation functions used in deep learning, connects their application trends with state-of-the-art research, and identifies directions for future comparison and research.

  • The paper provides a comprehensive summary of activation functions used in deep learning and highlights current application trends.
  • It introduces activation functions, categorizes the functions discussed, and identifies specific applications in deep learning architectures and systems.
  • Activation functions can improve pattern learning, automate feature detection, and support classification in neural networks.
  • Compounded activation functions represent an emerging direction in activation-function development.
  • The survey excludes some activation functions because it focuses on functions used in deep learning applications.
  • Future work should compare state-of-the-art functions on award-winning architectures using standard datasets to assess performance differences.
Loading 1811.03378v1…