Source-linked AI summary
Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark
Shiv Ram Dubey, Satish Kumar Singh, Bidyut Baran Chaudhuri
TL;DR
Activation functions are central to nonlinear feature learning, but existing choices differ in gradients, computation, nonlinearity, and adaptability. This survey organizes these functions, reviews their properties, and benchmarks 18 state-of-the-art choices across networks and data types. Its conclusions identify activation-function patterns associated with particular network architectures while noting limitations of several designs.
Problem
Existing activation functions have limitations involving diminishing gradients, limited nonlinearity, optimization difficulty, computational inefficiency, and lack of adaptability.
Method
The paper surveys and classifies activation functions, analyzes their properties, and compares 18 state-of-the-art functions across networks and four benchmark datasets with different modalities.
Results
Logistic Sigmoid and Tanh should be avoided for convolutional neural networks because they lead to poor convergence, whereas ReLU-family and related functions perform well in specified network settings.
Takeaways & Limitations
Activation-function suitability depends on the network and data setting, with ReLU remaining a common default while alternatives perform better for some applications.
Takeaways & Limitations
Logistic Sigmoid and Tanh suffer from vanishing gradients caused by saturating behavior, which makes deep-network training difficult.
Abstract
from arXiv · showhide
Neural networks have shown tremendous growth in recent years to solve numerous problems. Various types of neural networks have been introduced to deal with different types of problems. However, the main goal of any neural network is to transform the non-linearly separable input data into more linearly separable abstract features using a hierarchy of layers. These layers are combinations of linear and nonlinear functions. The most popular and common non-linearity layers are activation functions (AFs), such as Logistic Sigmoid, Tanh, ReLU, ELU, Swish and Mish. In this paper, a comprehensive overview and survey is presented for AFs in neural networks for deep learning. Different classes of AFs such as Logistic Sigmoid and Tanh based, ReLU based, ELU based, and Learning based are covered. Several characteristics of AFs such as output range, monotonicity, and smoothness are also pointed out. A performance comparison is also performed among 18 state-of-the-art AFs with different networks on different types of data. The insights of AFs are presented to benefit the researchers for doing further research and practitioners to select among different choices. The code used for experimental comparison is released at: \url{https://github.com/shivram1987/ActivationFunctions}.
1. Introduction
Activation functions enable neural networks to learn abstract features through nonlinear transformations. This survey classifies activation functions, analyzes their properties, and compares 18 state-of-the-art functions across networks and data modalities.
- Activation functions learn abstract features by introducing nonlinear transformations into neural networks.
- The survey classifies activation functions into Logistic Sigmoid/Tanh, Rectified Unit, Exponential Unit, Adaptive, and miscellaneous categories.
- It analyzes activation functions from multiple perspectives, including their characteristics, suitability for different data types, and recent deep-learning developments.
- The performance analysis compares 18 state-of-the-art activation functions using different networks on four benchmark datasets with different modalities.
- The survey provides insights intended to support further research and activation-function selection.
2. Evolution of Activation Functions
Activation functions evolved from linear and early Sigmoid/Tanh units toward ReLU, exponential, adaptive, and miscellaneous designs addressing limitations in nonlinearity, gradients, computation, and adaptability.
- Linear Activation Functions: Linear activation does not add nonlinearity, so multilayer networks remain linear despite having multiple layers.
- Logistic Sigmoid/Tanh Unit Based Activation Functions: Logistic Sigmoid squashes outputs to [0, 1], while Tanh squashes inputs to [−1, 1].
- Logistic Sigmoid/Tanh Unit Based Activation Functions: Sigmoid and Tanh suffer from saturation and vanishing gradients, making deep-network training difficult.
- Rectified Linear Unit Based Activation Functions: ReLU became widely used because of its simplicity and improved performance, while its variants address negative-value under-utilization, limited nonlinearity, and unbounded output.
- Exponential Unit Based Activation Functions: Exponential-unit activations use exponential functions to utilize negative values and address limitations associated with Sigmoid, Tanh, and ReLU.
- Learning/Adaptive Activation Functions: Learning-based adaptive activations introduce learnable parameters to adjust activation behavior to dataset complexity.
3. Logistic Sigmoid and Tanh Based AFs
This section reviews Logistic Sigmoid- and Tanh-based activation functions, including scaled, parametric, adaptive, and related variants. It emphasizes their output ranges, gradient behavior, and attempts to address saturation and vanishing gradients.
- Scaled and parametric variants: Scaled Hyperbolic Tangent (sTanh) addresses Tanh’s limited output range and zero-gradient problems by scaling its output to [−A, A].
- Scaled and parametric variants: Scaled Sigmoid and Penalized Tanh expand the available output ranges, but both continue to suffer from vanishing gradients.Their reported output ranges are [−2, 2] for sSigmoid and [−a, 1] for pTanh, where a ∈ (0, 1).
- Task-specific observations: pTanh performs better for Natural Language Processing tasks, although it retains the vanishing-gradient limitation.
- Other variants: Other variants target gradient flow or feature properties through added noise, scaled gradients, symmetry, boundedness, or learnable parameters.The noisy AF facilitates gradient flow in saturating regimes, while ReSech is differentiable, symmetric, and bounded; additional parameters can increase complexity.
- Improved sigmoid variants: Improved Logistic Sigmoid (ISigmoid) uses a piecewise combination of sigmoidal and linear functions to address Sigmoid’s vanishing-gradient problem.
4. Rectified Activation Functions
Rectified activation functions simplify computation but face limitations involving negative inputs, nonlinearity, and output bounds. Their variants modify slopes, thresholds, symmetry, or piecewise structure to address these issues.
- ReLU: ReLU outputs x for nonnegative inputs and zero otherwise, with range [0, ∞) and gradients of one and zero in the two regimes.Its simplicity reduces the computational complexity associated with Logistic Sigmoid and Tanh, but negative inputs have zero gradient.
- Negative-input utilization: Leaky ReLU uses a linear negative branch, while PReLU makes its negative-input slope trainable.LReLU requires choosing an appropriate negative slope, whereas PReLU can overfit because its slope is learned.
- Negative-input utilization: CReLU preserves information from original and negated inputs, while shifted and biased variants move the rectification point to capture negative values.Using multiple ReLUs or biases can increase model complexity.
- Adaptive and piecewise variants: Trainable and dynamic variants increase rectified-function flexibility through learnable slopes, multiple linear bins, or adaptive thresholds.Examples include Dynamic ReLU, SReLU, MTLU, and PLU; MTLU is nondifferentiable at multiple points, while PLU remains continuous but unbounded.
- Output and shape variants: vReLU, SignReLU, and related variants alter symmetry or negative-value handling, but vReLU still has a nonsymmetric output.The cited variants generally retain nonnegative output ranges, while other designs can extend outputs below zero.
- Summary: Rectified variants improve ReLU’s gradient diminishing, limited nonlinearity, and unboundedness issues, but do not resolve all ReLU limitations.The survey presents these as partial improvements rather than a complete solution.
5. Exponential Activation Functions
Exponential activation functions use exponential behavior to exploit negative inputs and address gradient diminishing in ReLU. Their variants adjust saturation, scaling, differentiability, or positive-side slopes.
- Core motivation: Exponential activation functions address ReLU’s gradient diminishing by incorporating negative inputs through exponential terms.ELU is the principal example discussed in this class.
- ELU variants: SELU adds a scaling hyperparameter that induces self-normalization toward zero mean and unit variance.The scaling makes the positive-side slope larger than one.
- Parametric variants: PELU, CELU, and MPELU make saturation, decay, positive slopes, or functional forms trainable or continuously differentiable.PELU uses trainable parameters a and b, while MPELU can represent rectified, exponential, or combined forms.
- ELU: ELU is differentiable, saturates for large negative inputs, reduces bias shift, and uses negative saturation to add noise robustness.Its output range is [−1, ∞).
- Efficiency and flexibility: FELU improves ELU efficiency using displacement bits and integer algebra operations, while EELU preserves a small non-zero negative gradient and an elastic positive slope.Both retain exponential-unit behavior while modifying computational or shape properties.
- Other variants: Other variants combine exponential units with sigmoid-like or ReLU-like components, including ELiSH, HardELiSH, PREU, RMAF, and ShELU.These designs differ in how they treat positive and negative input regions.
6. Learning/Adaptive Activation Functions
Learning and adaptive activation functions adjust nonlinear behavior using trainable parameters, learned bases, subnetworks, or combinations of existing activations. This flexibility adds parameters but can remain a small practical burden relative to the full network.
- Motivation: Adaptive activation functions address the inability of many fixed functions to adjust to dataset complexity.The survey identifies learning and adaptation as a recent trend.
- Parametric activations: APL learns different activation shapes for different neurons through trainable hinge-function parameters.Its output range is [0, ∞), and the number of hinges is a hyperparameter.
- Automatically searched activations: Swish uses a learnable β to adjust its shape between linear and ReLU-like behavior according to the learned nonlinearity.Smaller β values approach linear behavior, whereas higher values approach ReLU behavior.
- Combinations: Adaptive combinations blend functions such as PReLU and PELU or select and combine activation libraries using trainable weights.AAF combines multiple functions through a sigmoid-weighted expression, while ABU learns preferred activations.
- Learned representations: Learned activation shapes can also come from splines, Fourier expansions, hypernetworks, linear regression, or trainable parameters added to standard functions.These approaches range from nonparametric representations to neural networks that generate activation weights.
- Cost: Adaptive functions increase parameter count and complexity, but the added burden is reported as negligible relative to the entire network and reflected in training-time experiments.The survey refers to Table 10 for the experimental training-time comparison.
7. Miscellaneous Activation Functions
Miscellaneous activation functions include softplus-based, stochastic, polynomial, subnetwork, kernel, and complex-valued designs. These approaches trade smoothness, flexibility, noise handling, compactness, or computational cost in different ways.
- Softplus-based functions: Softplus-based activations provide smooth differentiability and include variants for statistical applications, softmax-like outputs, and spiking neural networks.Noisy Softplus is described as suitable for SNNs.
- Softplus-based functions: Rand Softplus stochastically mixes ReLU and Softplus using ρ, improving the network’s capability toward noise.The stochastic hyperparameter controls the mixture.
- Softplus-based functions: Mish combines Softplus with Tanh to produce a smooth, non-monotonic activation used in YOLOv4 for object detection.Its multiple functions increase complexity, which may limit use in deep networks.
- Stochastic functions: Stochastic activations include RReLU, EReLU, RTReLU, and GELU, but their sampling or probabilistic structure increases complexity and has limited exploration.GELU treats nonlinearity as a stochastic-regularization-driven transformation.
- Polynomial and rational functions: Polynomial, rational, and Padé-based activations learn or approximate common hand-designed functions, with RAF addressing PAU’s nonsmooth nature.PAUs can also learn new compactly represented activation functions.
- Subnetwork and kernel functions: Subnetwork and kernel-based functions increase flexibility through ensembles, dynamic linear functions, small neural networks, or inexpensive kernel expansions.KAF is extended to multikernel AFs, and related work considers complex-valued neural networks.
8. Aspects of Activation Functions
The section examines how activation functions affect optimization, network behavior, and suitability across data types. It also notes theoretical properties and the limited coverage of non-image data in prior work.
- Orthogonal initialization based on dynamical isometry can drastically accelerate network learning, while parameter initialization conditions can improve sigmoidal networks.
- ReLU networks can approximate smooth functions more efficiently when deep, while one-hidden-layer ReLU networks may reach global optima in polynomial time.
- ReLU-based networks can produce overconfident predictions far from the training data.
- Because most activation functions are evaluated on images, comparatively few studies examine their behavior on other data types.The survey uses Table 6 to summarize activation-function insights across networks and datasets.
9. Performance Comparison and Analysis
The paper compares 18 activation functions across image, text, and speech settings using multiple neural-network architectures. Results indicate that activation-function suitability depends on the network and data type, with some functions offering better accuracy or convergence but greater training cost.
- Comparison with Existing Survey/Performance Analysis: The survey covers more activation functions and broader network/data settings than existing surveys and performance analyses.
- Comparison with Existing Survey/Performance Analysis: The analysis evaluates 18 state-of-the-art activation functions across multiple neural networks and image, text, and speech data.The compared functions include traditional, ReLU-based, exponential, and recent adaptive functions.
- Experimental Performance Analysis: For image classification, Softplus, ELU, and CELU suit MobileNet, while ReLU, Mish, and PDELU perform well with VGG16, GoogLeNet, and DenseNet.
- Experimental Performance Analysis: ReLU, LReLU, ELU, GELU, CELU, ABReLU, and PDELU perform better with residual-connection networks.
- Experimental Performance Analysis: Sigmoid and Elliott show the poorest convergence in the SENet18 experiment, while PDELU is highly inefficient for CIFAR100 training time.
- Experimental Performance Analysis: Tanh and SELU are better suited for language translation, with PReLU, LiSHT, SRS, and PAU also performing better.
10. Conclusion and Recommendations
The conclusion organizes activation functions by their optimization benefits, limitations, and empirical suitability across architectures and data types. It recommends matching activation functions to network and dataset characteristics while balancing convergence, accuracy, and training cost.
- The survey provides an up-to-date classification and performance comparison of activation functions across image, text, and speech data.
- Sigmoid and Tanh improvements address non-zero-mean and zero-gradient problems but retain increased complexity.
- ReLU variants target negative-value under-utilization, limited nonlinearity, and unbounded output, but often do not outperform ReLU across models.
- Exponential activation functions improve negative-value utilization and saturation behavior, but many suffer from non-smoothness.
- Adaptive activation functions learn parameters for dataset-specific nonlinearity, but their performance depends on the base function, parameter count, and initialization.
- Activation-function choice should reflect model and dataset complexity to avoid overfitting or under-convergence.
- PDELU and SRS increase training-time complexity, whereas ReLU, SELU, GELU, and Softplus offer a promising accuracy–training-time tradeoff.
- Tanh and SELU suit language translation, while PReLU, GELU, Swish, Mish, and PAU are suggested for speech recognition.