Source-linked AI summary
Searching for Activation Functions
Prajit Ramachandran, Barret Zoph, Quoc V. Le
TL;DR
ReLU is effective but alternatives have not consistently displaced it across models and datasets. This paper searches automatically for scalar activation functions using exhaustive and reinforcement learning-based methods, then evaluates the best candidate, Swish. Swish consistently matches or outperforms ReLU, including ImageNet gains of 0.9% on Mobile NASNet-A and 0.6% on Inception-ResNet-v2, while remaining simple to substitute.
Problem
Hand-designed alternatives to ReLU have produced inconsistent gains across models and datasets, leaving ReLU the dominant activation function.
Method
The paper combines exhaustive and reinforcement learning-based search to discover scalar activation functions, then empirically evaluates the best candidate.
Results
Swish, f(x) = x · sigmoid(βx), consistently matches or outperforms ReLU across challenging domains, improving ImageNet top-1 accuracy by 0.9% on Mobile NASNet-A and 0.6% on Inception-ResNet-v2.
Takeaways & Limitations
Swish can replace ReLU with a simple one-line code change, even though the evaluated models and hyperparameters were designed for ReLU.
Takeaways & Limitations
The experimental results may not be directly comparable with corresponding prior results because the training setups differ.
Abstract
from arXiv · showhide
The choice of activation functions in deep networks has a significant effect on the training dynamics and task performance. Currently, the most successful and widely-used activation function is the Rectified Linear Unit (ReLU). Although various hand-designed alternatives to ReLU have been proposed, none have managed to replace it due to inconsistent gains. In this work, we propose to leverage automatic search techniques to discover new activation functions. Using a combination of exhaustive and reinforcement learning-based search, we discover multiple novel activation functions. We verify the effectiveness of the searches by conducting an empirical evaluation with the best discovered activation function. Our experiments show that the best discovered activation function, $f(x) = x \cdot \text{sigmoid}(βx)$, which we name Swish, tends to work better than ReLU on deeper models across a number of challenging datasets. For example, simply replacing ReLUs with Swish units improves top-1 classification accuracy on ImageNet by 0.9\% for Mobile NASNet-A and 0.6\% for Inception-ResNet-v2. The simplicity of Swish and its similarity to ReLU make it easy for practitioners to replace ReLUs with Swish units in any neural network.
1 INTRODUCTION
Activation functions strongly influence deep-network training, yet ReLU remains dominant because alternatives deliver inconsistent gains. This work uses automated search to discover scalar activation functions and evaluates the best candidate, Swish.
- ReLU remains the most successful and widely used activation function, enabling gradients to flow for positive inputs.Its simplicity and effectiveness have made it the default choice across deep learning.
- Hand-designed alternatives have not achieved ReLU’s widespread adoption because their performance gains are inconsistent across models and datasets.
- Automated exhaustive and reinforcement learning-based searches discover novel scalar activation functions that can replace ReLU without changing network architecture.
- Swish, defined as f(x) = x · sigmoid(βx), consistently matches or outperforms ReLU across challenging domains.On ImageNet, Swish improves top-1 classification accuracy by 0.9% on Mobile NASNet-A and 0.6% on Inception-ResNet-v2.
2 METHODS
The method searches a structured space of scalar activation functions built from repeated unary–binary core units. Exhaustive enumeration handles small spaces, while an RNN controller trained by reinforcement learning explores large spaces using child-network validation accuracy as reward.
- The search space composes unary and binary functions to balance candidate expressivity with search tractability.
- Each core unit applies two unary functions independently to scalar inputs and combines their outputs with a binary function.Repeated core units produce a scalar-to-scalar activation function.
- Exhaustive search is used for small spaces, whereas repeated core units create spaces on the order of 10^12 possibilities that require learned search.
- An autoregressive RNN controller predicts activation-function components sequentially and is trained with reinforcement learning for large search spaces.
- Candidate functions are evaluated by training child networks, recording validation accuracy, and using that accuracy to update or reward the search algorithm.Distributed workers parallelize child-network training to reduce wall-clock search time.
3 SEARCH FINDINGS
Searches conducted with small CIFAR-10 child networks produced activation functions that often generalized to larger architectures. The strongest patterns favored simple structures, raw preactivation access, and selected nonlinear forms, while Swish was chosen for broader evaluation.
- The search used ResNet-20 trained on CIFAR-10 for 10K steps, creating a risk that top functions would favor small networks.Additional experiments tested preactivation ResNet-164, Wide ResNet 28-10, and DenseNet 100-12.
- The best-performing activation functions were simple, typically representable with one or two core units.More complicated functions potentially underperformed because they were harder to optimize.
- Top activation functions commonly fed the raw preactivation x into the final binary function, a structure also used by ReLU.
- The searches identified periodic-function activation designs and found that division generally performed poorly near zero denominators.Periodic functions suggest a route for further research, while division worked when denominators were bounded away from zero or canceled by the numerator.
- Six of eight discovered activation functions generalized across architecture changes, and all six matched or outperformed ReLU on ResNet-164.
- Swish was selected for extensive evaluation because early experiments showed better generalization than max(x, σ(x)).
4 SWISH
Swish is a smooth, nonmonotonic activation whose behavior can be adjusted through β, interpolating between linear and ReLU-like functions. Its derivative and learned β values help explain its flexibility and effectiveness in deep networks.
- Definition and shape: Swish is defined as x · σ(βx), with β either fixed or trainable, and approaches ReLU-like behavior as β increases.At β = 0, it becomes x/2; as β →∞, the sigmoid component approaches a 0-1 function.
- Definition and shape: Unlike ReLU, Swish is smooth and nonmonotonic while remaining unbounded above and bounded below.Its distinctive nonmonotonic behavior appears as a bump for negative inputs.
- Derivative: Swish’s first derivative varies with β, and for β = 1 its magnitude is below 1 for inputs less than approximately 1.25.The derivative’s scale controls how quickly it approaches 0 and 1.
- Negative-input behavior: A large percentage of trained preactivations fall within Swish’s bump domain, −5 ≤ x ≤ 0, indicating that this negative-input behavior is important.The bump’s shape can be adjusted through β.
- Negative-input behavior: Trained β values in Mobile NASNet-A range from 0 to 1.5 and peak near β ≈ 1, suggesting that models use trainable β flexibility.Fixing β = 1 is effective in practice, while training β can further improve some models.
5 EXPERIMENTS WITH SWISH
The experiments benchmark Swish against ReLU and other activation functions across image classification and machine translation. Swish matches or exceeds baselines on nearly all tasks, with especially strong gains on mobile ImageNet models.
- Swish matches or exceeds baseline activation functions on nearly all tasks across multiple models and datasets.The aggregate comparison includes CIFAR, ImageNet, and English→German translation, with statistical significance under a one-sided paired sign test.
- Experimental setup: The evaluation compares Swish with trainable β and fixed β = 1, and results may not be directly comparable with corresponding prior works.The experiments use multiple model architectures and datasets, with some baseline-specific training adjustments for convergence.
- CIFAR: Swish and Swish-1 consistently match or outperform ReLU on every evaluated CIFAR-10 and CIFAR-100 model.Swish also matches or exceeds the best baseline on almost every model, although the best baseline varies by model.
- ImageNet: 0.5%: Swish outperforms ReLU by this amount on Inception-ResNet-v2 ImageNet classification.Swish performs especially well on mobile-sized models, while gains are more limited on Inception-v4.
- ImageNet: 1.4% and 2.2%: Swish boosts top-1 ImageNet accuracy over ReLU on Mobile NASNet-A and MobileNet, respectively.Swish matches or exceeds the best baseline on most models; switching to Swish generally improves performance with little additional tuning.
- Machine translation: Swish outperforms or matches other baselines on WMT English→German machine translation.Swish-1 exceeds the next-best baseline by 0.6 BLEU points on newstest2016, while Softplus is the worst-performing baseline across this domain.
6 RELATED WORK
Related work applies search techniques to discover traditionally hand-designed neural-network components, including architectures, optimizers, and activation functions. This work focuses on scalar activation functions while positioning the study within meta-learning and broader activation-function research.
- Search techniques have been used to discover convolutional and recurrent architectures as well as optimizers.The paper frames such automated discovery as part of the revived subfield of meta-learning.
- Prior studies have used meta-learning to find initializations for one-shot learning and adaptable reinforcement learning.The related work also includes meta-learning applications beyond architecture and optimizer discovery.
- This work studies scalar activation functions that transform one scalar into another scalar, unlike many-to-one pooling or gating functions.The paper distinguishes scalar functions from broader classes of nonlinear functions used in deep networks.
- Activation-function research has largely proposed new functions, while relatively few studies systematically compare alternatives.The paper identifies its comparison of scalar activation functions across multiple challenging datasets as novel to the authors’ knowledge.
- Swish consistently outperforms ReLU on deep models, challenging conventional assumptions about ReLU’s gradient-preserving property.The paper argues that residual connections can enable optimization of very deep networks, reducing the need for individual components to preserve gradients.
7 CONCLUSION
The paper uses automatic search to discover activation functions and validates Swish as a strong ReLU replacement. Swish consistently outperforms ReLU even when ReLU-designed models and hyperparameters are used unchanged.
- Automatic search discovered novel activation functions with strong empirical performance, including the activation function named Swish.Swish is defined as f(x) = x · sigmoid(βx).
- Swish consistently outperformed ReLU and other activation functions across the experiments.The evaluation simply replaced ReLU activations in models and retained hyperparameters designed for ReLU.
- Using Swish with ReLU-designed models and hyperparameters produced gains despite being a simple, suboptimal replacement procedure.The authors expect additional gains from specifically designing models and hyperparameters for Swish.
- Replacing ReLUs with Swish units requires only a simple one-line code change in any network.This practicality follows from Swish’s simplicity and similarity to ReLU.