Source-linked AI summary
Mish: A Self Regularized Non-Monotonic Activation Function
Diganta Misra
TL;DR
Activation functions strongly influence neural-network performance and training, while ReLU can discard gradient information for negative inputs. The paper proposes Mish, a self-regularized non-monotonic activation, and evaluates it against established alternatives across vision benchmarks. Mish generally matches or improves competing activations, although the authors identify computational overhead and unresolved questions about its regularizing mechanism.
Problem
ReLU can lose gradient information by collapsing negative inputs to zero, motivating activation functions that improve performance while addressing this weakness.
Method
The paper proposes Mish, defined as f(x) = xtanh(softplus(x)), and evaluates it against ReLU, Leaky ReLU, and Swish across neural-network architectures and computer-vision tasks.
Results
Mish generally matches or improves Swish, ReLU, and Leaky ReLU across the evaluated computer-vision benchmarks.
Takeaways & Limitations
Mish provides a broadly competitive activation function whose smooth, non-monotonic behavior is associated with consistent empirical improvements under the tested conditions.
Takeaways & Limitations
The authors identify further Mish-CUDA optimization, broader model and task evaluation, and clarification of the derivative-based regularizing mechanism as future work.
Abstract
from arXiv · showhide
We propose $\textit{Mish}$, a novel self-regularized non-monotonic activation function which can be mathematically defined as: $f(x)=x\tanh(softplus(x))$. As activation functions play a crucial role in the performance and training dynamics in neural networks, we validated experimentally on several well-known benchmarks against the best combinations of architectures and activation functions. We also observe that data augmentation techniques have a favorable effect on benchmarks like ImageNet-1k and MS-COCO across multiple architectures. For example, Mish outperformed Leaky ReLU on YOLOv4 with a CSP-DarkNet-53 backbone on average precision ($AP_{50}^{val}$) by 2.1$\%$ in MS-COCO object detection and ReLU on ResNet-50 on ImageNet-1k in Top-1 accuracy by $\approx$1$\%$ while keeping all other network parameters and hyperparameters constant. Furthermore, we explore the mathematical formulation of Mish in relation with the Swish family of functions and propose an intuitive understanding on how the first derivative behavior may be acting as a regularizer helping the optimization of deep neural networks. Code is publicly available at https://github.com/digantamisra98/Mish.
1 Introduction
Activation functions shape neural-network nonlinearity, performance, and training dynamics, but ReLU can lose gradient information on negative inputs. The paper proposes Mish, a self-regularized non-monotonic alternative inspired by Swish, and reports matching or improved performance across computer-vision tasks.
- Motivation: Activation functions introduce nonlinearity to layer inputs, and their choice affects neural-network performance.The layer operation is z = g(∑i w_i x_i + b).
- Motivation: ReLU improved generalization and convergence speed over Sigmoid and TanH but can lose gradient information by collapsing negative inputs to zero.This failure mode is commonly called Dying ReLU.
- Activation-function design: Mish and Swish have distinctive negative concavity unlike ReLU, preserving small negative weights.The figure also compares their first and second derivatives.
- Contribution: Mish is a self-regularized non-monotonic activation inspired by Swish’s self-gating property and defined as f(x) = xtanh(softplus(x)).The paper evaluates it against Swish, ReLU, and Leaky ReLU across computer-vision tasks.
2 Motivation
The paper motivates Mish through activation functions that preserve negative weights and maintain smooth, non-monotonic profiles, then validates candidate functions on CIFAR-10. It further links Mish’s derivative behavior to smoother optimization and reports more stable or consistently better behavior than alternatives.
- Design motivation: Swish-like activation functions are studied for non-monotonicity, preservation of small negative weights, and smooth profiles.Mish was developed through systematic analysis and experimentation over properties associated with Swish’s effectiveness.
- CIFAR-10 validation: A six-layer CNN trained on CIFAR-10 for three 50-epoch RMSProp runs found Mish performed better than the validated experimental functions.The candidate xlog(1+tanh(e^x)) was comparable but often unstable and divergent in deeper architectures.
- Mathematical relation: Mish’s first derivative is presented as closely related to Swish’s derivative, with softplus(x) = ln(1+e^x) and sigmoid(x) = 1/(1+e^-x).The equation comparison is used to analyze Mish’s relation to the Swish family.
- Optimization interpretation: The authors speculate that Mish’s Δ(x) parameter acts like a preconditioner that smooths gradients and may ease optimization.They describe this interpretation as based on experimental observations.
- Activation properties: Mish is non-monotonic, smooth, and preserves small negative weights, properties associated with consistent improvement over Swish in deep networks.These properties are presented as accounting for Mish’s performance and improvement when substituted for Swish.
- Output landscapes: Mish produces a smoother output landscape than ReLU in a five-layer randomly initialized network, whereas ReLU shows sharper transitions.The paper connects smoother output landscapes with smoother loss landscapes and easier optimization.
3 Mish
Mish is presented as a smooth, continuous, self-regularized, non-monotonic activation function whose properties are associated with improved gradient flow and optimization landscapes. Experiments compare it with ReLU and Swish across loss landscapes, depth, noise, and initialization settings.
- Mish is a smooth, continuous, self-regularized, non-monotonic activation function with a range of [≈-0.31, ∞).It is bounded below and unbounded above.
- Mish preserves small negative information, avoiding by design the preconditions necessary for the Dying ReLU phenomenon.The paper links this property to better expressivity and information flow.
- Mish produces smoother output landscapes than ReLU, which the paper associates with smoother loss landscapes and easier optimization.The ReLU landscape has sharp transitions, whereas Mish has a smoother profile.
- Mish yielded a smoother, better-conditioned ResNet-20 loss landscape and the lowest loss compared with ReLU and Swish after 200 CIFAR-10 training epochs.The paper also reports that Mish had a wider minimum, while ReLU had multiple local minima.
- Mish consistently achieved better loss than ReLU and Swish as additive Gaussian noise intensity increased in MNIST inputs.The experiment used a five-layer convolutional network optimized with SGD.
- Mish showed a consistent positive performance difference over Swish across Glorot, LeCun normal, and He uniform variance scaling initializers.This comparison used a six-layer convolutional network on CIFAR-10.
4 Benchmarks
Mish is evaluated across image classification and object detection benchmarks, generally matching or exceeding ReLU, Swish, and Leaky ReLU while preserving constant network settings in key comparisons. The benchmarks also examine statistical consistency, data augmentation, and implementation efficiency.
- CIFAR-10: Mish outperformed other activation functions on CIFAR-10 using SqueezeNet across 23 runs, achieving the highest mean accuracy, second-lowest mean loss, and third-lowest accuracy deviation.The experiments used Adam for 50 epochs while changing only the activation function.
- CIFAR-10: Mish consistently outperformed ReLU and Swish across standard CIFAR-10 architectures, often improving performance by 1% to 3% over ReLU-enabled networks.The comparison swapped activation functions while keeping other network and training parameters constant.
- ImageNet-1k: Mish exceeded default Leaky ReLU or ReLU on all four reported ImageNet architectures, including a 1% Top-1 accuracy increase over Leaky ReLU on CSP-ResNet-50.In CSP-ResNext-50, Swish decreased Top-1 accuracy by 13.4% versus Leaky ReLU, whereas Mish improved it by 1%.
- MS-COCO Object Detection: Replacing ReLU with Mish improved mAP@0.5 by 0.4% on CSP-DarkNet-53 and CSP-DarkNet-53+PANet+SPP object detectors.For CSP-DarkNet-53, the reported mAP@0.5 was 65.7% at 65 FPS on a Tesla V100.
- MS-COCO Object Detection: Mish improved YOLOv4 AP50val by 0.9% to 2.1% across three variants using CSP-DarkNet-53 and CSP-PANet backbones at test size 736.The evaluated variants were YOLOv4pacsp, YOLOv4pacsp-s, and YOLOv4pacsp-x.
- Efficiency: Mish-CUDA reduced Mish’s computational overhead, maintained baseline learning performance without stability issues, and was faster than native PyTorch Softplus in single precision.The runtime comparison covered forward and backward passes in FP16 and FP32 on an NVIDIA GeForce RTX-2070 GPU over 100 runs.
5 Conclusion
The paper concludes that Mish generally improves empirical results over Swish, ReLU, and Leaky ReLU while retaining related smooth, non-monotonic properties. It also identifies computational overhead, broader evaluation, and theoretical understanding of its regularizing term as future work.
- Mish provides better empirical results than Swish, ReLU, and Leaky ReLU under most experimental conditions.The paper notes that Mish shares several properties with Swish and GELU, including an unbounded positive domain, bounded negative domain, non-monotonic shape, and smooth derivative.
- Data augmentation techniques such as CutMix and Label Smoothing behave consistently with expectations alongside Mish experiments.
- Future work includes reducing Mish-CUDA computational overhead and evaluating Mish in additional state-of-the-art computer-vision models and tasks.The authors also propose obtaining a normalizing constant that could reduce dependence on Batch Normalization layers.
- The contribution of the Δ(x) term in Mish’s first derivative remains theoretically important to investigate as a possible regularizing mechanism.Understanding the behavior and conditions governing this term could support more principled activation-function design.