Source-linked AI summary
A survey on modern trainable activation functions
Andrea Apicella, Francesco Donnarumma, Francesco Isgrò, Roberto Prevete
TL;DR
Neural-network research has proposed many trainable activation functions, but their terminology and properties are heterogeneous. The paper surveys and taxonomizes these approaches, finding that many can be represented as deeper fixed-activation networks with constrained parameters, while noting expressivity and parameter-overhead limitations in some families.
Problem
The literature contains heterogeneous trainable activation-function models and inconsistent use of the term “activation function,” motivating a systematic survey and classification.
Method
The paper surveys feed-forward approaches, analyzes their properties, and proposes a taxonomy based on how activation functions are defined and whether their shape is trainable.
Results
Many proposed trainable activation functions are equivalent to adding neuron layers with fixed activation functions and simple constraints on corresponding weights.
Takeaways & Limitations
Trainable activation functions can often be understood as subnetworks within deeper architectures using fixed activation functions.
Takeaways & Limitations
Parameterized standard functions may remain close in shape to their fixed counterparts, limiting expressivity; some models also increase parameters with hidden-unit count.
Abstract
from arXiv · showhide
In neural networks literature, there is a strong interest in identifying and defining activation functions which can improve neural network performance. In recent years there has been a renovated interest of the scientific community in investigating activation functions which can be trained during the learning process, usually referred to as "trainable", "learnable" or "adaptable" activation functions. They appear to lead to better network performance. Diverse and heterogeneous models of trainable activation function have been proposed in the literature. In this paper, we present a survey of these models. Starting from a discussion on the use of the term "activation function" in literature, we propose a taxonomy of trainable activation functions, highlight common and distinctive proprieties of recent and past models, and discuss main advantages and limitations of this type of approach. We show that many of the proposed approaches are equivalent to adding neuron layers which use fixed (non-trainable) activation functions and some simple local rule that constraints the corresponding weight layers.
1 Introduction
The paper surveys trainable activation functions, clarifies inconsistent terminology, and organizes heterogeneous approaches while showing that many correspond to deeper fixed-activation networks with constrained parameters.
- Motivation: Trainable activation functions learn their shape during network training alongside parameters such as weights and biases.The approach seeks an appropriate function shape from information in the training data.
- Survey scope: The survey examines feed-forward approaches, isolates their critical properties, proposes a definition-based taxonomy, and discusses expressivity and performance.Many approaches can also be applied to recurrent neural networks.
- Main finding: Many trainable activation approaches are equivalent to adding fixed-activation neuron layers with simple local constraints on corresponding weights.Equivalent constraints may include fixing or sharing weights and arranging layers appropriately.
- Terminology: The literature uses “activation function,” “transfer function,” and “output function” inconsistently, sometimes assigning them distinct roles.One formalization distinguishes internal activation, scalar output, and their composition as the transfer function.
- Terminology: Some proposed models alter the entire neuron computation rather than applying a simple nonlinearity to a linear input combination.The paper suggests calling these models trainable transfer functions when preserving the activation/output distinction.
2 A taxonomy of activation functions
The proposed taxonomy first separates fixed-shape from trainable-shape functions, then divides trainable functions into parameterized standard functions and ensemble-based families.
- Primary classification: The taxonomy’s primary distinction is whether the activation-function shape can change during training.Fixed-shape functions include sigmoid, tanh, and ReLU; trainable-shape functions learn their shape from training data.
- Fixed-shape functions: Fixed-shape functions are divided into classic activations and rectified-based functions, including sigmoid, tanh, step, and ReLU examples.The paper identifies rectified functions as a turning point associated with improved network performance and renewed research interest.
- Trainable-shape functions: Trainable activation functions are grouped into parameterized standard functions and functions based on ensemble methods.Parameterized standard functions add trainable parameters to a standard fixed activation, allowing its shape to be tuned from data.
- Ensemble methods: The ensemble-method family includes a subclass called linear combination of one-to-one functions, which mixes distinct one-variable functions.These component functions may themselves contain additional parameters.
- Subnetwork interpretation: Many ensemble-based and analytic approaches can be represented as single-input subnetworks nested within the main network architecture.This representation supports analysis of their implementation and relationship to deeper networks.
3 Fixed-shape activation functions
Fixed-shape activation functions are defined without parameters changed during training and include classic functions, ReLU-family variants, and parameterized forms based on fixed functions. ReLU-based functions address deep-network training problems, while several variants introduce trade-offs or limited expressiveness.
- Scope and organization: Fixed-shape activation functions have no parameters modified during training and are commonly used as building blocks for trainable activation functions.The section surveys these functions before discussing trainable combinations and variations.
- ReLU and its properties: ReLU became standard in modern neural-network architectures because rectification helps avoid vanishing gradients, although it can suffer from dying units and non-differentiability at zero.ReLU also facilitates sparse coding, but large negative biases can force outputs to remain zero.
- Rectifier-based variants: Leaky ReLU introduces a small negative-region gradient to address inactive ReLU units, but empirical results found nearly identical performance to standard rectifiers.Its negative-side slope is fixed at 0.01, while randomized Leaky ReLU samples the slope from a uniform distribution.
- Rectifier-based variants: Softplus smoothly approximates ReLU, yet experiments suggested ReLU properties support supervised training better than the smoother alternative.The comparison concerns the expected training benefit of smoothness and differentiability.
- Rectifier-based variants: ELU retains the identity for positive inputs and uses nonzero negative outputs, potentially speeding learning while requiring the additional hyper-parameter α.The negative outputs push activation means closer to zero.
- Parameterized standard activations: Parameterized standard activations tune the shapes of fixed functions, but their learned forms remain similar to the originals and provide only a poor increase in expressiveness.The survey gives AGSig, AGTanh, and Swish as examples of this pattern.
4 Trainable Activation functions
The survey organizes trainable activation functions into parameterized standard functions and ensemble methods, then examines their representations, training requirements, and limitations. It shows that many such functions can be represented as shallow subnetworks using fixed activations and constrained parameters.
- Taxonomy: Trainable activation functions are grouped into parameterized standard functions and activation functions based on ensemble methods.Parameterized functions tune a shape resembling a fixed activation, whereas ensemble methods merge different functions.
- Parameterized standard activation functions: Parameterized standard activations tune a fixed-function shape with trainable parameters, requiring corresponding changes to gradient-based learning.The added parameters require partial derivatives during training; examples include generalized sigmoid, generalized hyperbolic tangent, sigmoidal selector, and Swish.
- Parameterized standard activation functions: Swish is parameterized by α, approaches ReLU as α →+∞, and equals SiLU when α = 1.Its shape remains related to the fixed functions on which it is based.
- Limitations: The surveyed approaches have limitations including restricted shape expressiveness, specialized backpropagation formulas, parameter overhead, extra optimization costs, and hyper-parameter requirements.Some parameterized functions remain close to their base shapes; APL overhead is 2 · k · n, and interpolation-based methods require additional tuning.
- Neural-network equivalence: Many trainable activation functions can be modeled by shallow subnetworks of neurons with fixed activations and simple parameter constraints.For Swish, the trainable parameter can be represented by a learned connection weight while another connection is constrained; similar constructions apply to other functions, including SReLU.
- Ensemble methods: Activation-function search can construct functions by composing unary and binary basis functions, with search performed over specified composition spaces.Reported methods include search techniques and genetic algorithms for learning activation functions.
5 Trainable non-standard neuron definitions
The survey treats Maxout, NIN, CIC, and related models as trainable non-standard neuron definitions because they alter neuron computation rather than only the scalar activation shape. These approaches can improve flexibility or performance, but often add parameters and may depart from single-neuron activation functions.
- Maxout: Maxout units compute multiple trainable linear responses and return their maximum, rather than applying an activation function to one weighted sum.The resulting unit is a piece-wise linear approximator capable of approximating any convex function.
- Maxout: Maxout can generalize rectifier-based units and was reported to improve classification errors over ReLU in cross-validation experiments.The improvement comes with substantially higher learning cost because each neuron uses k times as many parameters.
- Maxout: Multi-layer Maxout composes Maxout functions and replaces only part of a traditional network’s activations to trade accuracy against computational resources.The composition further increases the number of parameters, motivating partial rather than universal replacement.
- NIN and CIC: NIN replaces convolution-layer activations with multilayer perceptrons applied to input patches, while CIC uses a sparse MLP instead of a fully connected one.NIN produces channel-wise outputs from the final MLP layer, but can require many extra parameters and does not constrain the final layer to one output.
- NIN and CIC: Batch-Normalized Maxout NIN replaces NIN’s ReLU functions with Maxout and adds Batch Normalization to address zero saturation and data-distribution changes.
- Local competition: LWTA blocks use local competition so only the winning neuron propagates its output, selecting different parameter subsets for different inputs.The authors hypothesize that input-dependent parameter subsets can improve learning accuracy.
- Taxonomy: The survey excludes Maxout from trainable activation functions because its maximum operator is the activation of a multi-response neuron, classifying it instead as a trainable non-standard neuron definition.It is equivalent to a standard shallow network with k hidden identity-activated neurons and a maximum-output neuron.
6 Performance and experimental architecture comparison
Reviewed studies generally report higher accuracy for trainable activations, but cross-study comparison is difficult because architectures, algorithms, hyperparameters, and datasets vary. The survey therefore organizes these experimental setups and notes that apparent gains may reflect increased architectural complexity.
- Different neural-network architectures, learning algorithms, hyperparameters, and datasets make exhaustive comparison across proposed approaches difficult.
- Reviewed studies report accuracy improvements over non-trainable activation functions, including comparisons with ReLU, sigmoid, and other trainable models.
- Performance improvements may reflect increased complexity in the final setup architecture rather than the activation function’s learning capability alone.
- Table 2 catalogs the architectures and datasets used in experiments to sketch the variation underlying reported comparisons.
7 Conclusions
The survey classifies trainable activation functions into a taxonomy spanning fixed-shape and trainable-shape functions, while reviewing their reported performance and architectural interpretations. It further observes that several trainable functions can be represented through deeper fixed-activation subnetworks with constrained weights.
- The proposed taxonomy divides activation functions into fixed-shape and trainable-shape categories.
- Trainable-shape functions include parameterized standard functions, ensemble methods, and linear combinations of one-to-one functions.
- Several reviewed works report substantial performance improvements over equivalent architectures using classical fixed-shape activations such as ReLU or sigmoid.
- Several trainable activations are linear combinations of fixed nonlinear functions and can therefore be represented as nested subnetworks.
- Equivalent deeper architectures can use fixed-shape activations with constraints such as weight sharing instead of trainable activation functions.