Source-linked AI summary
Activate or Not: Learning Customized Activation
Ningning Ma, Xiangyu Zhang, Ming Liu, Jian Sun
TL;DR
Activation functions such as Swish remain poorly understood, motivating more effective and general alternatives. The paper introduces ACON and meta-ACON, which learn whether to activate neurons, and reports gains across models and tasks, including 6.7% and 1.8% ImageNet top-1 improvements on MobileNet-0.25 and ResNet-152.
Problem
Despite Swish’s widespread use, its mechanism remains poorly understood, motivating investigation of more effective activation functions.
Method
ACON approximates Maxout-family activations with smooth differentiable functions, while meta-ACON learns activation switching through learnable bounds.
Results
ACON transfers across classification, object detection, and semantic segmentation, improving ImageNet top-1 accuracy by 6.7% on MobileNet-0.25 and 1.8% on ResNet-152.
Takeaways & Limitations
ACON provides an effective activation alternative across small and large models and a design space for layer-wise, channel-wise, or pixel-wise activation switching.
Takeaways & Limitations
Exploring more complicated functions in the Maxout family is beyond this work’s scope and remains a future direction.
Abstract
from arXiv · showhide
We present a simple, effective, and general activation function we term ACON which learns to activate the neurons or not. Interestingly, we find Swish, the recent popular NAS-searched activation, can be interpreted as a smooth approximation to ReLU. Intuitively, in the same way, we approximate the more general Maxout family to our novel ACON family, which remarkably improves the performance and makes Swish a special case of ACON. Next, we present meta-ACON, which explicitly learns to optimize the parameter switching between non-linear (activate) and linear (inactivate) and provides a new design space. By simply changing the activation function, we show its effectiveness on both small models and highly optimized large models (e.g. it improves the ImageNet top-1 accuracy rate by 6.7% and 1.8% on MobileNet-0.25 and ResNet-152, respectively). Moreover, our novel ACON can be naturally transferred to object detection and semantic segmentation, showing that ACON is an effective alternative in a variety of tasks. Code is available at https://github.com/nmaac/acon.
1. Introduction
The paper introduces ACON by interpreting Swish as a smooth ReLU approximation and extending this perspective to a broader, learnable activation family. Meta-ACON explicitly learns neuron activation decisions, yielding stable gains across model scales and vision tasks.
- Motivation: Swish is represented as a smooth approximation to ReLU, providing a perspective for understanding its NAS-searched performance.The paper motivates this interpretation because Swish remains widely used but poorly understood.
- ACON: ACON generalizes the ReLU–Swish conversion by approximating Maxout with smooth, differentiable ACON-A, ACON-B, and ACON-C functions, with Swish as ACON-A.The approach is conceptually simple and adds no computational overhead.
- Meta-ACON: Meta-ACON explicitly optimizes a switching factor that learns whether neurons should activate, addressing gradient descent’s difficulty learning linear-to-nonlinear switching.The switching factor decays toward zero as the nonlinear function becomes linear.
- Meta-ACON: Meta-ACON provides a new design space for layer-wise, channel-wise, or pixel-wise activation decisions and improves diverse tasks, including highly optimized deep models.The paper describes these improvements as stable across various tasks and model settings.
- Results: 6.7% and 1.8% ImageNet top-1 accuracy improvements over ReLU are achieved on MobileNet-0.25 and ResNet-152, respectively, with transfer to detection and segmentation.The gains hold for both small and large models, and ACON generalizes beyond image classification.
2. Related Work
Prior activation-function work spans widely used ReLU variants, smooth approximations, and the general Maxout formulation, while dynamic networks adapt computation to inputs. ACON extends these directions by smoothly approximating Maxout and dynamically learning the activation’s nonlinear degree as a new design space.
- Activation functions: ReLU and its variants are among the most widely used activation functions, while softplus provides a smooth LogSumExp-based approximation to ReLU.ReLU is non-differentiable at zero but differentiable elsewhere.
- Activation functions: Maxout generalizes ReLU and leaky ReLU, approximates linear activations, and motivates ACON as a smooth approximation to the Maxout family.Swish is cited as a searched scalar activation in this context.
- Activation functions: DY-ReLU adapts piecewise-linear activations using global context but adds many parameters, with gains that become smaller in larger and deeper models.The cited example reports ResNet50-DY-ReLU improving from 76.2 to 77.2, a 1.0% improvement.
- Dynamic network: Dynamic CNNs condition kernels, widths, depths, or other structural choices on input samples, whereas ACON dynamically learns the activation function’s nonlinear degree.This nonlinear degree controls how strongly the nonlinear layer is applied.
- Neural network design space: Neural-network architecture design spaces commonly optimize feature dimensions, feature combinations, kernel shape, and kernel computation; ACON adds the activation’s nonlinear degree as another design dimension.The passage distinguishes feature-level and kernel-level design spaces.
3. ACON
ACON approximates Maxout-style activations with a smooth switching factor that learns whether neurons should behave nonlinearly or linearly. Its general ACON-C form enables learnable gradient bounds, while meta-ACON explicitly conditions the switching factor on the input through a flexible design space.
- ACON-A and ACON-B: For ReLU, ACON-A becomes x · σ(βx), exactly recovering Swish and interpreting Swish as a smooth approximation to ReLU.The same smooth-maximum conversion extends to other Maxout-family activations such as PReLU through ACON-B.
- ACON-C: ACON-C generalizes ACON-B by applying distinct channel-wise scaling parameters p1 and p2 to the two arguments of the smooth maximum.The paper initializes β=p1=1 and p2=0, with all three parameters channel-wise.
- ACON: ACON uses a switching factor β to interpolate between nonlinear maximum behavior as β →∞ and linear arithmetic-mean behavior as β →0.This lets individual neurons adaptively activate or not.
- Meta-ACON: Meta-ACON learns β explicitly as an input-conditioned function G(x), allowing layer-wise, channel-wise, or pixel-wise switching-factor designs.The experiments use a channel-wise structure with ACON-C unless otherwise noted.
4. Experiment
Experiments on ImageNet show that ACON and meta-ACON substantially improve classification accuracy across lightweight and deep CNNs, while meta-ACON also outperforms SENet and generalizes to semantic segmentation. Ablations attribute these gains to smooth activation transitions and sample-adaptive switching factors.
- Meta-ACON evaluation: Meta-ACON improves accuracy by 6.7% on MobileNetV1 0.25, around 3% on 300M-level models, and 2.0% and 1.8% on ResNet-50 and ResNet-101.These results are reported on ImageNet with 224x224 inputs and compare against ReLU baselines.
- ACON variants: ACON-A, ACON-B, and ACON-C outperform their max-based functions, while ACON-C surpasses ACON-A (Swish) and ACON-B.ACON-C also provides continuous gains of 1.0% on ResNet-101, compared with 0.1% for ACON-A (Swish).
- Comparison with other activations: Against other activations, Swish improves the top-1 error rate by 1.1%, whereas ACON and meta-ACON improve it by 2.4% and 4.6%.The comparison includes ReLU, Swish, Mish, ELU, and SoftPlus on ShuffleNetV2 0.5x.
- Switching factor distribution: Meta-ACON learns sample-specific switching factors with broader distributions than ACON, causing some neurons to remain close to inactive while others use higher non-linearity.The learned factor can vary by sample and can be designed at pixel-wise, channel-wise, or layer-wise levels.
- Comparison with SENet: Meta-ACON improves ResNet-152 accuracy by 1.8% and gains 1% over SENet, outperforming SENet across all evaluated network structures.The comparison covers lightweight CNNs and deep CNNs under matched evaluation settings.
- Semantic segmentation: On Cityscapes semantic segmentation, PSPNet with a ResNet-50 backbone reaches 78.3, 1.1 points above the ReLU baseline and better than Swish.The experiment uses the CityScapes dataset and compares activation choices within the same segmentation framework.
5. Conclusion
The paper concludes that ACON is a simple, effective activation that learns whether to activate neurons, while approximating the general Maxout family. It identifies broader exploration of Maxout functions and applying ACON across more applications as future directions.
- 5. Conclusion: ACON is presented as a simple, effective activation that learns whether to activate neurons and approximates the general Maxout family.The authors characterize ACON as the central contribution of the work.
- 5. Conclusion: Exploring more functions in the Maxout family and applying ACON across a wide range of applications are identified as future directions.The authors describe broader Maxout exploration as beyond this work’s focus and expect ACON to transfer broadly.
Appendix
The appendix reports ShuffleNetV2 training curves on ImageNet and specifies the TFNet building block for stride-1 and stride-2 operation.
- Training curves: ShuffleNetV2 training curves are shown for ImageNet.
- TFNet building block: For TFNet downsampling, ACON-FReLU2 uses max pooling for ηa(x) and stride-2 depth-wise convolution for ηb(x).The building block is illustrated for stride=1 and stride=2.