Source-linked AI summary
Deep Learning with S-shaped Rectified Linear Activation Units
Xiaojie Jin, Chunyan Xu, Jiashi Feng, Yunchao Wei, Junjun Xiong, Shuicheng Yan
TL;DR
Existing activation functions have limited ability to model different function shapes. SReLU uses learnable piecewise-linear functions inspired by psychophysical laws, and experiments show improved deep-network performance with negligible parameter increases.
Problem
Existing activation functions have limited ability to learn both convex and non-convex functions, while maxout also introduces substantial parameter and training costs.
Method
SReLU uses three piecewise linear functions with four learnable parameters, trained jointly with the network and initialized through a freezing method.
Results
Experiments on CIFAR-10, CIFAR-100, MNIST, and ImageNet with NIN and GoogLeNet show that SReLU effectively boosts deep-network performance compared with other activation functions.
Takeaways & Limitations
SReLU can be used across existing deep networks while learning both convex and non-convex functions with negligible additional parameters.
Takeaways & Limitations
Maxout cannot learn non-convex functions and introduces extra parameters that increase storage and training costs in very deep CNNs.
Abstract
from arXiv · showhide
Rectified linear activation units are important components for state-of-the-art deep convolutional networks. In this paper, we propose a novel S-shaped rectified linear activation unit (SReLU) to learn both convex and non-convex functions, imitating the multiple function forms given by the two fundamental laws, namely the Webner-Fechner law and the Stevens law, in psychophysics and neural sciences. Specifically, SReLU consists of three piecewise linear functions, which are formulated by four learnable parameters. The SReLU is learned jointly with the training of the whole deep network through back propagation. During the training phase, to initialize SReLU in different layers, we propose a "freezing" method to degenerate SReLU into a predefined leaky rectified linear unit in the initial several training epochs and then adaptively learn the good initial values. SReLU can be universally used in the existing deep networks with negligible additional parameters and computation cost. Experiments with two popular CNN architectures, Network in Network and GoogLeNet on scale-various benchmarks including CIFAR10, CIFAR100, MNIST and ImageNet demonstrate that SReLU achieves remarkable improvement compared to other activation functions.
Introduction
Existing activation functions improve CNN optimization but have limited ability to learn non-convex transformations. SReLU addresses this gap by using learnable piecewise-linear forms inspired by psychophysical laws.
- Activation-function limitations: ReLU, LReLU, PReLU, and maxout are essentially convex and cannot learn non-convex functions.APL can approximate non-convex functions but imposes a restrictive unit-slope, zero-bias constraint on its rightmost linear function.
- Motivation: SReLU is a novel activation unit inspired by the Webner-Fechner and Stevens laws in psychophysics and neural sciences.These laws describe logarithmic and power-function relationships between stimulus intensity and perceived strength.
- Proposed method: SReLU uses three piecewise linear functions with four learnable parameters to model both convex and non-convex transformations.Its parameterized forms imitate the logarithmic and power-function forms associated with the two psychophysical laws.
- Proposed method: Because SReLU uses non-saturated piecewise-linear functions, it retains fast convergence and avoids exploding or vanishing gradients.The paper identifies these as advantages shared with other non-saturated activation functions.
- Evaluation: Experiments with Network in Network and GoogLeNet across CIFAR10, CIFAR100, MNIST, and ImageNet report remarkable improvement over other activation functions.The evaluation spans benchmarks with different scales and tests SReLU in two popular CNN architectures.
Related Work
The related work surveys piecewise-linear activation units and highlights limitations in their ability to represent non-convex functions or avoid restrictive parameter constraints. It then motivates SReLU as a piecewise-linear approach inspired by psychophysical laws to approximate both convex and non-convex functions.
- Activation units: ReLU, LReLU, PReLU, APL and maxout are reviewed as piecewise-linear activation units used in deep networks.The section introduces these activation functions before discussing psychophysical laws and SReLU.
- Limitations: APL sums hinge-shared linear functions but explicitly constrains the rightmost line to unit slope and zero bias.The authors argue that downstream nonlinear or unrestorable operations may prevent this lost freedom from being recovered.
- Limitations: Maxout computes the largest output among multiple linear functions and can approximate any convex function in theory.Its K linear functions introduce extra parameters, increasing storage and training-time costs in very deep CNNs.
- Motivation: The Webner-Fechner and Stevens laws relate physical stimulus magnitude to perceived intensity and motivate SReLU's function forms.SReLU imitates logarithmic and power functions with piecewise-linear functions to approximate non-linear convex and non-convex functions.
S-shaped Rectified Linear Units (SReLU)
SReLU combines three piecewise linear functions with four learnable parameters, enabling unconstrained approximation of both convex and non-convex functions. It is trained jointly with the network and initialized through freezing and data-adaptive threshold selection.
- Definition: SReLU combines three linear functions into an activation unit with four learnable parameters that can vary across channels.Its parameters include thresholds and slopes for the piecewise regions.
- Function forms: SReLU can approximate both convex and non-convex functions without constraining its learnable parameters.Its flexible rightmost slope also allows adaptive scaling for large-magnitude inputs, unlike APL’s fixed output behavior there.
- Parameterization: Channel-wise SReLU adds 4N parameters, including 21.7K in GoogLeNet, while keeping memory size and computation time almost unchanged.The channel-wise variant performs slightly better than the channel-shared variant on CIFAR-10 without data augmentation.
- Training: SReLU is jointly trained with the deep network by gradient descent and back propagation.Parameter gradients are derived by the chain rule, with feature-map contributions summed over relevant positions and channels for shared variants.
- Initialization: The freezing initialization first degenerates SReLU into a conventional LReLU, then adapts thresholds using input distributions from the training data.This addresses layer-dependent input magnitudes and provides a better starting point for optimization while allowing reuse of pretrained LReLU models.
- Comparison with Other Activation Functions: SReLU includes ReLU, LReLU, and PReLU as special cases and outperforms APL by 0.98% on CIFAR-10 and 3.04% on CIFAR-100 in NIN experiments.The comparison used no data augmentation or the proposed initialization strategy.
Experiments and Analysis
Experiments evaluate SReLU across multiple datasets and CNN architectures, using controlled replacements of ReLU and standard network configurations. Across CIFAR-10, CIFAR-100, MNIST, and ImageNet, SReLU generally outperforms competing activations with small parameter overhead.
- Experimental setup: Experiments use NIN on CIFAR-10, CIFAR-100, and MNIST, and GoogLeNet on ImageNet.The evaluation spans datasets with different scales and follows the original networks' published configurations.
- CIFAR results: SReLU achieves the best performance among the compared methods on CIFAR-10 and CIFAR-100.The comparisons include ReLU, LReLU, PReLU, APL, maxout, and other deep-network methods.
- CIFAR results: 1.26%, 1.34%, and 1.33% error reductions over ReLU, LReLU, and PReLU, respectively, are reported on CIFAR-10 without data augmentation.On CIFAR-100, the corresponding reductions are 4.86%, 4.90%, and 4.85%.
- Parameter cost: SReLU adds only 5.68K parameters to the 0.97M-parameter original NIN in the CIFAR experiments.The paper reports this increase as slight relative to the original network.
- Additional benchmarks: SReLU consistently outperforms other methods on augmented CIFAR-10 and CIFAR-100 and performs better than other methods on MNIST.CIFAR augmentation uses random horizontal reflection, while MNIST is evaluated without data augmentation.
- ImageNet results: 1.24% improvement on ImageNet is obtained over the original GoogLeNet using ReLU, with only 21.6K additional parameters.ImageNet tests use a single model and single view; the original GoogLeNet has 5M parameters.
Conclusion
The paper proposes SReLU as a flexible activation unit for deep networks and reports performance improvements across four datasets and two CNN architectures. It concludes that SReLU can learn both convex and non-convex functions and be used in existing networks.
- Conclusion: SReLU is proposed as an activation unit inspired by fundamental laws in psychophysics and neural sciences.The paper evaluates it in deep networks rather than limiting the proposal to a specific architecture.
- Conclusion: SReLU can learn both convex and non-convex functions and can be used in existing deep networks.The conclusion presents these properties as advantages over other activation functions.
- Conclusion: Experiments on CIFAR-10, CIFAR-100, MNIST, and ImageNet with NIN and GoogLeNet show that SReLU boosts deep-network performance.The paper identifies applications beyond vision, such as NLP, as future work.