PaperlayerView exact revision on arXiv

Source-linked AI summary

How important are activation functions in regression and classification? A survey, performance comparison, and future directions

Ameya D. Jagtap, George Em Karniadakis

arXiv:2209.02681v6cs.LG

TL;DR

Activation functions are important to neural-network learning, yet the paper notes the range of proposed functions and their advantages and limitations across tasks. It surveys fixed and adaptive functions, taxonomies, and comparisons in classification and physics-informed machine learning. The reported comparisons find adaptive and Rowdy activations perform well on PDE tasks, while JAX yields substantially higher predictive accuracy than TensorFlow and PyTorch in those comparisons.

  • Problem

    Activation functions have been proposed for classification and regression, but their advantages and limitations vary across functions and tasks.

    Source: B. Contributions of this work · p. 2

  • Method

    The paper surveys fixed and adaptive activations, proposes an applications-based taxonomy, and compares functions in classification and physics-informed machine-learning settings.

    Source: B. Contributions of this work · p. 2, §XI · p. 18

  • Results

    For PDEs, adaptive and Rowdy activations perform well on clean and Gaussian-noisy data, and JAX improves predictive accuracy by an order of magnitude over TensorFlow and PyTorch for all activations.

    Source: §XII · p. 22

  • Takeaways & Limitations

    Physics-informed machine-learning activation choices must account for accurate evaluation of spatio-temporal derivatives to enforce physical laws.

    Source: §XI · p. 18, §XII · p. 22

    For physics-informed problems requiring derivatives, step function, ReLU, and leaky ReLU are poor candidates because their derivatives do not exist at particular locations.

    Source: A. Differentiability · p. 18

Abstract

from arXiv · show

Inspired by biological neurons, the activation functions play an essential part in the learning process of any artificial neural network commonly used in many real-world problems. Various activation functions have been proposed in the literature for classification as well as regression tasks. In this work, we survey the activation functions that have been employed in the past as well as the current state-of-the-art. In particular, we present various developments in activation functions over the years and the advantages as well as disadvantages or limitations of these activation functions. We also discuss classical (fixed) activation functions, including rectifier units, and adaptive activation functions. In addition to discussing the taxonomy of activation functions based on characterization, a taxonomy of activation functions based on applications is presented. To this end, the systematic comparison of various fixed and adaptive activation functions is performed for classification data sets such as the MNIST, CIFAR-10, and CIFAR- 100. In recent years, a physics-informed machine learning framework has emerged for solving problems related to scientific computations. For this purpose, we also discuss various requirements for activation functions that have been used in the physics-informed machine learning framework. Furthermore, various comparisons are made among different fixed and adaptive activation functions using various machine learning libraries such as TensorFlow, Pytorch, and JAX.

I. INTRODUCTION

Activation functions add nonlinearity to artificial neurons, helping networks learn complex features, but no single function is optimal for every problem. This survey reviews fixed and adaptive functions, their properties and applications, and comparisons across classification and physics-informed machine-learning settings.

B. Contributions of this work

  • The survey covers classical fixed activations, including rectifier units, and their properties, alongside modern adaptive activation functions.It also reviews oscillatory and non-standard classical activations.Source: B. Contributions of this work · p. 2
  • The authors propose an applications-based taxonomy spanning real- and complex-valued activations and discuss quantized activations for network efficiency.The taxonomy is framed for scientific computation.Source: B. Contributions of this work · p. 2
  • The survey compares fixed and adaptive activations on MNIST, CIFAR-10, and CIFAR-100, and evaluates activations for PDEs across TensorFlow, PyTorch, and JAX.The physics-informed comparisons use clean and noisy data and include library run times.Source: B. Contributions of this work · p. 2

A. Aim

  • Activation functions introduce nonlinearity that helps neural networks learn complex data features; without it, a network acts as a linear regression model.The passage cites Cybenko and Hornik on conditions for universal approximation.Source: §II · p. 2

II. ACTIVATION FUNCTIONS: A HISTORICAL PERSPECTIVE

  • The historical account traces sigmoid and tanh use, their saturation-related vanishing gradients, and the later popularity of ReLU and its variants.It also notes ReLU’s nondifferentiability at zero, unboundedness, and dying-ReLU problem.Source: §II · pp. 2–3

III. BIOLOGICAL VS ARTIFICIAL NEURONS

  • Artificial neurons combine weights and bias, an activation function, and an output; unlike biological neurons, they send continuous values.The activation function changes artificial-neuron firing behavior.Source: §III · p. 4

IV. DESIRED CHARACTERISTICS OF THE ACTIVATION FUNCTIONS

  • Desirable activation properties include nonlinearity, low computational cost, gradient behavior, boundedness, and differentiability, with trade-offs across functions.Non-saturating activations are discussed as one remedy for vanishing gradients.Source: §IV · p. 4

V. TAXONOMY OF ACTIVATION FUNCTIONS

The paper organizes activation functions by both their characteristics and their applications. It then describes common fixed functions, outlining their forms and properties.

A. Characterization based taxonomy

  • Activation functions are grouped as fixed, adaptive, or non-standard; adaptive functions include parametric and ensemble approaches, while non-standard functions may be fixed or adaptive.The taxonomy also distinguishes linear and nonlinear functions.Source: A. Characterization based taxonomy · p. 5, Figure 3 · p. 5

B. Application based taxonomy

  • The application-based taxonomy distinguishes real- and complex-valued activations and includes quantized activations for computational efficiency.Complex-valued outputs occur in scientific and engineering applications; quantized outputs are integers and can reduce memory needs.Source: B. Application based taxonomy · p. 5, Figure 4 · p. 5

A. Linear and piece-wise linear functions

  • Linear activation has a constant gradient and an unbounded range, while the piece-wise linear form has range [0, 1] and is nondifferentiable at x = ±b.The piece-wise form’s derivative is zero outside the interval bounded by ±b.Source: A. Linear and piece-wise linear… · p. 5

B. Step function

  • The step function, also called the Heaviside function, outputs 0 below zero and 1 at or above zero, with an undefined derivative at zero.Its derivative is zero elsewhere, and its range is {0, 1}.Source: B. Step function · pp. 5–6

C. Sigmoid function

  • Sigmoid maps inputs to [0,1] and is bounded, but saturation and non-zero-centered outputs are among its drawbacks; deep networks trained with it proved less effective.The passage also describes improved logistic sigmoid as proposed to address vanishing gradients.Source: C. Sigmoid function · p. 6
  • The section surveys sigmoid improvements, including bipolar-sigmoid, Elliott, and scaled sigmoid [87]; bipolar sigmoid has range [-1,1] and Elliott has range [0,1].Source: C. Sigmoid function · p. 6

D. Hyperbolic tangent (tanh) function

Tanh was widely used in the late 1990s and early 2000s and is generally used for regression. Its zero-centered output is an advantage, while saturation can cause vanishing gradients.

D. Hyperbolic tangent (tanh) function

  • Tanh was widely used in the late 1990s and early 2000s, is generally used for regression, and has a zero-centered range of [−1, 1].Its saturation region makes learning faster difficult and causes the vanishing-gradient problem.Source: D. Hyperbolic tangent (tanh) function · p. 6
  • The section surveys additional proposed tanh variants, including rectified hyperbolic secant, Hexpo, and penalized hyperbolic tangent.Hexpo has range [−c, a].Source: D. Hyperbolic tangent (tanh) function · p. 6
  • The linearly scaled tanh function LiSHT, Φ(x) = x tanh(x), scales tanh to tackle its gradient-diminishing problem.The section also describes scaled tanh, whose amplitude A sets its output range to [−A, A].Source: D. Hyperbolic tangent (tanh) function · p. 6

E. Rectified Linear Unit (ReLU)

ReLU is widely used for classification, is easy to implement, and was introduced to address vanishing gradients. Its zero output for negative inputs can leave neurons inactive, motivating a range of variants.

E. Rectified Linear Unit (ReLU)

  • ReLU’s zero outputs for negative inputs can stop neurons responding to gradient-based optimization, especially at high learning rates.The survey calls this the dead or dying ReLU problem.Source: E. Rectified Linear Unit (ReLU) · p. 7
  • Leaky ReLU adds a small negative-side slope to address dying neurons, but its leakage parameter α must be chosen appropriately.The survey reports that α = 0.01 is used in most cases and cites a comparison concluding that Leaky ReLU always outperforms ReLU.Source: E. Rectified Linear Unit (ReLU) · p. 7
  • CReLU has range [0, ∞), while its disadvantage relative to ReLU is increased model complexity.Source: E. Rectified Linear Unit (ReLU) · p. 7
  • Elastic ReLU uses a random number, and randomly translated ReLU shifts its input by a Gaussian-sampled α before applying a zero threshold.Source: E. Rectified Linear Unit (ReLU) · p. 7
  • Other variants alter ReLU’s shape or outputs: bounded ReLU caps its output, while DReLU generalizes ReLU and SReLU through a movable inflection.Bounded ReLU’s maximum output is A; DReLU becomes ReLU at δ = 0 and SReLU at δ = 1.Source: E. Rectified Linear Unit (ReLU) · p. 7
  • Natural-Logarithm ReLU applies a logarithmic modification to positive inputs to increase nonlinearity, with output range 0 to ∞.Its β is constant.Source: E. Rectified Linear Unit (ReLU) · p. 7

F. Gaussian Error Linear Units (GELUs)

The survey describes GELU, Softplus, and ELU alongside their properties and variants. GELU uses an expected stochastic transformation, Softplus smooths ReLU, and ELU has nonzero negative-side outputs.

F. Gaussian Error Linear Units (GELUs)

  • GELU takes the expected value of multiplying input x by a Bernoulli variable whose probability is given by a Gaussian cumulative distribution.The survey gives a tanh approximation and notes that SGELU was developed for bidirectional convergence.Source: F. Gaussian Error Linear Units (GELUs) · p. 7

G. Softplus function

  • Softplus is an infinitely differentiable, smooth approximation to ReLU; noisy Softplus is described as matching LIF neuron responses and improving spiking-network performance.For noisy Softplus, x is the mean input, σ sets noise level, and k controls curve scaling.Source: G. Softplus function · p. 8

H. Exponential linear unit

  • ELU was reported to outperform ReLU variants with reduced training time and better test accuracy.Its negative outputs and nonzero negative-side gradient help move average output toward zero and avoid dead neurons; at α = 1 it is smooth everywhere.Source: H. Exponential linear unit · p. 8
  • SELU was reported to make a stack of dense layers self-normalizing when all hidden layers use SELU, subject to conditions.Source: H. Exponential linear unit · p. 8

I. Mish function

The survey spans Mish, radial and wavelet activations, oscillatory functions, and non-standard units. It relates activation properties and structures to tasks such as classification, signal identification, XOR learning, and probability output.

I. Mish function

  • Mish is self-gated and non-monotonic, with C∞ continuity, unbounded outputs above, and bounded outputs below.Its first derivative can be expressed using Swish.Source: §I · p. 9

J. Radial activation functions

  • Gaussian radial basis activations have been applied to approximation and classification, and were reported to outperform hardlim and sign on UCI datasets using RVFL networks.Source: J. Radial activation functions · p. 9

K. Wavelet activation functions

  • Wavelet neural networks have been used in classification and identification, and can identify important features in high-frequency signals.The survey also describes Gaussian-wavelet activations as piecewise-linear and non-monotone.Source: K. Wavelet activation functions · p. 9

L. Oscillatory activation functions

  • Oscillatory activations can provide multiple zeros, enabling a single neuron to distinguish XOR classes that require two decision hyperplanes.Rowdy activation injects adaptable oscillatory noise over a monotonic base as optimization starts, creating multiple hypersurfaces.Source: L. Oscillatory activation functions · pp. 9–10

M. Non-standard activation functions

  • Maxout takes the maximum of k linear feature mappings and is designed for use with dropout; Softmax normalizes outputs into a probability distribution.The survey describes Maxout’s subspace pooling as helping prevent units from remaining idle.Source: M. Non-standard activation functions · p. 10

VII. COMPLEX-VALUED ACTIVATION FUNCTIONS

The survey reviews complex-valued activations and quantized activations, whose uses and constraints differ. Complex activation design faces a boundedness–analyticity constraint, while quantized activations can improve efficiency but introduce gradient mismatch.

VII. COMPLEX-VALUED ACTIVATION FUNCTIONS

  • A bounded complex activation that is also complex-differentiable is not feasible, so many CVNN approaches use bounded nonanalytic split activations.Split activations apply real-valued functions separately to real and imaginary parts.Source: §VII · pp. 10–11
  • Analytic bounded activations with singular points require careful scaling of inputs and initial weights to avoid those points during training.Source: §VII · p. 11

VIII. QUANTIZED ACTIVATION FUNCTIONS

  • Quantizing activations can save memory and support faster training, but fixed-point activations create gradient mismatch because the effective activation is nondifferentiable.The mismatch reflects a discrepancy between assumed and real activation functions.Source: §VIII · pp. 11–12
  • Wide reduced-precision networks compensated for quantization precision loss by increasing the number of filters in each layer.The cited work also found activations take up more memory than weights.Source: §VIII · p. 12
  • For quantized ReLU, straight-through estimation treats the function as identity inside clipping bounds and constant outside, yielding unit and zero derivatives respectively.Source: §VIII · p. 12

IX. A QUEST TOWARDS AN OPTIMAL ACTIVATION

The survey frames activation choice as problem-dependent, with no general rule for selecting an optimal function. It reviews adaptive approaches that tune parameters or combine functions, alongside reported benefits and methods for learning activation shapes.

IX. A QUEST TOWARDS AN OPTIMAL ACTIVATION

  • Adaptive activations can introduce trainable parameters or combine predefined functions, and may outperform fixed counterparts of the same type.The survey presents these approaches as responses to the lack of a general rule for choosing an optimal activation.Source: §IX · p. 12

A. Parametric Activation Functions

  • Parametric activation designs include generalized hyperbolic tangent parameters, spline-based architectures, and learned per-neuron amplitudes.The survey also reviews trainable-amplitude functions, parametrically learned functions, and evolutionary approaches to finding activation functions.Source: A. Parametric Activation Functions · pp. 12–13
  • Swish is reported to converge faster and generalize better on many test problems; it is unbounded above and bounded below.Its tuning parameter is β; the survey also introduces E-swish as a proposed variant.Source: A. Parametric Activation Functions · p. 12
  • Locally adaptive activations assign trainable parameters per hidden layer or per neuron, with a slope-recovery term also described.The survey reports that layer-wise and neuron-wise versions can be trained faster.Source: A. Parametric Activation Functions · p. 12

1) Adaptive family of rectifier and exponential units:

Adaptive rectifier and exponential-unit families modify fixed activations through trainable slopes, thresholds, shifts, or piecewise parameters. The surveyed variants span alternative negative-input behavior, smoothness, and multi-bin linear forms.

1) Adaptive family of rectifier and exponential units:

  • Parametric ReLU learns its negative-side slope, while the surveyed adaptive family includes additional rectifier and exponential-unit variants.The survey lists Parametric ELU, S-shaped ReLU, Flexible ReLU, and Paired ReLU among these variants.Source: 1) Adaptive family of rectifier and… · p. 13
  • Continuously Differentiable ELU modifies ELU's negative values so its derivative exists and equals 1 at x = 0 for every α.Fast ELU is described as an approximation consistent with ELU's curve trend that preserves its accuracy advantage.Source: 1) Adaptive family of rectifier and… · pp. 13–14
  • Flexible ReLU captures negative values with a rectified point and has range [b, ∞), while Parametric Tanh Linear Unit has range [−α, ∞).Both α and β are trainable for Parametric Tanh Linear Unit.Source: 1) Adaptive family of rectifier and… · pp. 13–14
  • Paired ReLU uses trainable thresholds with scale parameters initialized to 0.5 and −0.5, and S-shaped ReLU uses four learnable parameters that can vary by channel.The survey describes S-shaped ReLU as a combination of three linear functions.Source: 1) Adaptive family of rectifier and… · pp. 13–14
  • Multi-bin Trainable Linear Units use trainable slopes, intercepts, and bin boundaries to form a piecewise-linear function with range (−∞, ∞).The survey also includes Mexican ReLU, Lipschitz ReLU, and parametric deformable ELU among the adaptive family.Source: 1) Adaptive family of rectifier and… · p. 14
  • Elastic ELU is designed to combine properties of ELU and ReLU, while parametric ELU learns α during training to set activation shape at each CNN layer.The survey also gives a multiple parametric ELU form with an additional β parameter on negative inputs.Source: 1) Adaptive family of rectifier and… · pp. 13–14

B. Stochastic/probabilistic adaptive activation functions

Stochastic adaptive activations add noise or perturbations to activation outputs, including in saturated regimes. The survey also describes probabilistic activations that retain drawbacks of the activation they generalize.

B. Stochastic/probabilistic adaptive activation functions

  • Noisy activations add structured, bounded noise and learn its level in saturated regimes, where the output is pinned to threshold t before noise is added.The noise is represented as s = μ + σξ; ξ is independently and identically distributed, and μ and σ define a location-scale family.Source: B. Stochastic/probabilistic adaptive… · p. 14
  • ProbAct is both trainable and stochastic, with perturbation z = σϵ; because it generalizes ReLU, it has the same drawbacks as ReLU.Its mean may be static or learnable, and ϵ is sampled from a normal distribution N(0, 1).Source: B. Stochastic/probabilistic adaptive… · pp. 14–15
  • Rand Softplus models stochasticity-adaptability using a stochastic hyperparameter ρ.Source: B. Stochastic/probabilistic adaptive… · p. 15

C. Fractional adaptive activation functions

Fractional adaptive activations use a fractional derivative parameter to generate families of activation functions. Ensemble approaches instead combine or select functions, with Kronecker Neural Networks providing a framework that includes several familiar adaptive and fixed network forms.

C. Fractional adaptive activation functions

  • Fractional activation functions add tunable behavior through Mittag-Leffler-type functions and fractional-calculus derivatives, producing families indexed by a.The survey describes replacing the exponential function with standard activations such as tanh, sigmoid, and logistic.Source: C. Fractional adaptive activation… · p. 15

D. Ensemble adaptive activation functions

  • Ensemble activations combine or select functions, including sigmoid, radial-basis, and sinusoid functions, or combinations chosen through discrete optimization.Other surveyed approaches use trained variables to construct activations, reinforcement learning, or data-driven combinations of basic activations.Source: D. Ensemble adaptive activation… · p. 15

C. Fractional adaptive activation functions

D. Ensemble adaptive activation functions

  • A polynomial activation of degree P can theoretically approximate any smooth function, but its global coefficient influence can make outputs excessively large.The survey also reviews kernel-based non-parametric activations and adaptive blending units.Source: D. Ensemble adaptive activation… · pp. 15–16
  • Kronecker Neural Networks vary parameters and K to represent different adaptive activations, including standard feed-forward, PReLU, ELU, SELU, and self-learnable activation networks.The survey describes KNN as a general framework that can generalize a class of feed-forward networks using adaptive activations.Source: D. Ensemble adaptive activation… · p. 16

X. PERFORMANCE OF SOME FIXED AND ADAPTIVE ACTIVATION FUNCTIONS FOR CLASSIFICATION TASKS

The section compares fixed and adaptive activation functions on three image-classification datasets, then turns to activation-function requirements for physics-informed machine learning. Across the classification datasets, adaptive and rowdy activations outperform fixed alternatives.

X. PERFORMANCE OF SOME FIXED AND ADAPTIVE ACTIVATION FUNCTIONS FOR CLASSIFICATION TASKS

  • The classification comparison uses literature accuracy reports and evaluates MobileNet and VGG16 on MNIST, CIFAR-10, and CIFAR-100.The reported literature results do not account for the architectures used in those experiments.Source: §X · p. 16

popular machine learning library.

  • Across MNIST, CIFAR-10, and CIFAR-100, adaptive and rowdy activation functions outperform; almost all activation functions perform well on MNIST.Leaky ReLU, ELU, and Sine work well on CIFAR-10 and CIFAR-100. Accuracy comparisons report means and standard deviations over 10 realizations.Source: popular machine learning library · pp. 17–18

XI. ACTIVATION FUNCTIONS FOR PHYSICS-INFORMED MACHINE LEARNING

  • Physics-informed machine learning uses automatic differentiation to incorporate governing physical laws, and its activation functions must support relevant spatial and temporal derivatives.The section investigates these requirements using physics-informed neural networks, which combine governing equations, experimental data, and initial or boundary conditions in a loss function.Source: §XI · p. 18

A. Differentiability

For physics-informed neural networks, activation functions must support derivatives used to enforce physical laws, and their output range can encode prior knowledge about field variables. Across three PDE cases, sine, tanh, and Swish perform well, while derivative and range requirements constrain activation choice.

A. Differentiability

  • Automatic differentiation computes network-output derivatives for the ODE residual, so the activation function affects how the network enforces physical laws.The network is trained against both data mismatch and the differential-equation residual.Source: A. Differentiability · p. 18, Figure 13 · p. 18
  • Differential-equation problems impose high-order continuity requirements, making step, ReLU, and leaky ReLU poor choices where their derivatives do not exist.The comparison considers PDE cases whose derivative order increases across cases.Source: A. Differentiability · pp. 18–19
  • Sine, tanh, and Swish perform well across all three PDE cases; ELU performs poorly in the first and cannot be used in the other two because its higher-order derivative does not exist.Sigmoid performs poorly, especially for PDEs involving high-order derivatives, while adaptive and rowdy activations perform well compared with fixed counterparts.Source: A. Differentiability · p. 19

B. Range

  • When a field variable's range is known, an output activation can encode that prior—for example, enforcing positive density when no vacuum state is present.The passage states that this activation choice and prior information have an immediate impact on training convergence.Source: B. Range · p. 19

C. Performance comparison using different machine learning libraries

The comparisons show JAX improving predictive accuracy over TensorFlow and PyTorch for Boussinesq PDE solutions, while adaptive and Rowdy activations add runtime cost. The summary also contrasts activation behavior across classification and regression and outlines directions for reducing adaptive-activation costs.

C. Performance comparison using different machine learning libraries

  • JAX improves predictive accuracy over TensorFlow and PyTorch for clean and noisy Boussinesq data across the tested activation functions.With 5% Gaussian noise, accuracy degrades by almost an order of magnitude, while JAX still offers about an order-of-magnitude improvement over the other libraries.Source: C. Performance comparison using… · p. 20, Table VI · p. 20, Table VII · p. 21
  • JAX with JIT is computationally more efficient, while adaptive activations add 8–10% runtime and Rowdy activations add about 75–80% versus fixed activations.The runtime comparison covers the first 10,000 Adam iterations for the Boussinesq equation with clean data.Source: C. Performance comparison using… · p. 21, Figure 15 · p. 22
  • Rowdy activations perform much better than fixed and adaptive activations in the tested discontinuous-function approximation and Helmholtz equation cases.The comparisons use 10,000 Adam iterations for function approximation and 40,000 for the Helmholtz equation.Source: C. Performance comparison using… · p. 21, Table VIII · p. 23

XII. SUMMARY

  • Quantized activations are described as more efficient than full-precision counterparts with similar accuracy, and as a route to handling large models and data in edge computing.The survey discusses quantization for both real- and complex-valued activations.Source: §XII · pp. 21–22
  • Adaptive activations are more expensive but can outperform fixed activations; on MNIST nearly all tested functions perform well, while swish, ReLU, Leaky ReLU, ELU, sine, adaptive, and Rowdy activations perform well on CIFAR-10 and CIFAR-100.The comparisons use MobileNet and VGG16 architectures.Source: §XII · p. 22
  • ReLU and its variants achieve state-of-the-art performance in many image-classification problems, but ReLU and ELU do not perform well for PIML regression because their derivatives do not exist.Sine, hyperbolic tangent, and swish perform well for regression problems.Source: §XII · pp. 21, 23

A. Some future directions

  • Future directions include quantized adaptive activations to mitigate cost, influential adaptive parameters that reshape the loss landscape, and mapping LIF parameters to ANN activations.The authors describe the LIF-to-activation mapping as challenging and propose it as a direction toward more efficient and universal activations.Source: A. Some future directions · p. 23

Related papers

Loading 2209.02681v6…