Source-linked AI summary
Empirical Evaluation of Rectified Activations in Convolutional Network
Bing Xu, Naiyan Wang, Tianqi Chen, Mu Li
TL;DR
The paper asks whether ReLU’s sparsity explains its performance and evaluates alternative rectified activations in convolutional networks. Across experiments, modified leaky ReLUs consistently outperform ReLU, with RReLU particularly favorable on small datasets.
Problem
The paper examines whether ReLU’s sparse activations drive performance and whether alternative non-saturated activations can outperform it.
Method
The authors empirically compare ReLU, Leaky ReLU, PReLU, and randomized Leaky ReLU across network architectures and datasets.
Results
Three modified leaky ReLUs consistently outperform ReLU across the evaluated datasets, while very leaky ReLU performs better than ReLU.
Takeaways & Limitations
On small datasets, randomized Leaky ReLU is favorable because training randomness reduces the risk of overfitting.
Takeaways & Limitations
How these activations perform on large-scale data remains an open question requiring further investigation.
Abstract
from arXiv · showhide
In this paper we investigate the performance of different types of rectified activation functions in convolutional neural network: standard rectified linear unit (ReLU), leaky rectified linear unit (Leaky ReLU), parametric rectified linear unit (PReLU) and a new randomized leaky rectified linear units (RReLU). We evaluate these activation function on standard image classification task. Our experiments suggest that incorporating a non-zero slope for negative part in rectified activation units could consistently improve the results. Thus our findings are negative on the common belief that sparsity is the key of good performance in ReLU. Moreover, on small scale dataset, using deterministic negative slope or learning it are both prone to overfitting. They are not as effective as using their randomized counterpart. By using RReLU, we achieved 75.68\% accuracy on CIFAR-100 test set without multiple test or ensemble.
1. Introduction
The paper challenges the belief that ReLU’s superior performance comes from activation sparsity by empirically comparing ReLU with leaky variants. It finds that leaky activations consistently outperform ReLU on small datasets, with RReLU favored because training randomness reduces overfitting.
- Background: Non-saturated activations such as ReLU help address exploding or vanishing gradients and accelerate convergence.The introduction contrasts them with saturated activations such as sigmoid and tanh.
- Motivation: ReLU prunes negative inputs to zero, producing sparse activations that are commonly believed to explain its superior performance.ReLU retains the positive part while setting the negative part to zero.
- Activation Functions: Leaky ReLU assigns a non-zero negative slope, while PReLU learns that slope from data.These functions broaden the rectified-unit family beyond ReLU’s complete removal of negative inputs.
- Activation Functions: RReLU randomizes negative slopes within a given range during training and fixes them during testing.Its randomized nature was reported to reduce overfitting in a recent Kaggle competition.
- Empirical Findings: On small datasets, Leaky ReLU and its variants consistently outperform ReLU in convolutional neural networks.The paper states that larger datasets require further investigation.
2. Rectified Units
This section defines four rectified units—ReLU, Leaky ReLU, PReLU, and RReLU—and distinguishes them by whether their negative-region parameter is fixed, learned, or randomized. RReLU samples its training slope from a uniform distribution and uses its training average during testing.
- Overview: The paper introduces four rectified units: ReLU, Leaky ReLU, PReLU, and RReLU.These units are illustrated for comparison and formally introduced in subsequent subsections.
- Leaky ReLU: Leaky ReLU experiments use ai = 5.5 in addition to the large value suggested in the original work, such as 100.The passage states that ai is a fixed parameter in range (1, +∞).
- Parameterization: PReLU learns ai during training via back propagation, whereas Leaky ReLU uses a fixed ai.PReLU is described as equivalent to Leaky ReLU except for how ai is determined.
- RReLU: RReLU samples aji as a random number from U(l, u) during training.This randomized slope is the defining feature of RReLU relative to Leaky ReLU.
- RReLU: During testing, RReLU sets aji to the average of its training values; this paper samples aji from U(3, 8).The test-time average is l+u 2, following the passage’s description of the dropout-style procedure.
3. Experiment Settings
The experiments compare activation functions under matched convolutional-network settings on CIFAR datasets, with additional evaluation on CIFAR-100 Inception and plankton classification. CIFAR results use raw images and single-view testing without augmentation or ensembles.
- Activation-function comparison: Classification performance is compared across activation functions using two state-of-the-art convolutional architectures with shared hyperparameters, trained in CXXNET2.This controls the activation-function comparison despite the large parameter-search space.
- CIFAR datasets: CIFAR-10 and CIFAR-100 contain 32x32 RGB images in 10 and 100 classes, respectively, with 50,000 training and 10,000 test images each.The experiments use raw images without preprocessing or augmentation.
- CIFAR architecture: The baseline CIFAR network is adapted from Network in Network, with an activation function following each convolutional layer.The network structure is specified in Table 1.
- CIFAR-100 Inception experiment: 75.68% test accuracy was achieved by RReLU on a subset of the Batch Norm Inception Network in CIFAR-100 without ensemble or multiple-view testing.The evaluated Inception subset starts from the inception-3a module.
- Plankton classification: The National Data Science Bowl task classifies 30,336 labeled grayscale plankton images across 121 classes, using 25,000 for training and 5,336 for validation.The private test set contains 130,400 images, and performance is evaluated with multi-class log-loss.
4. Result and Discussion
Across CIFAR-10, CIFAR-100, and NDSB experiments, all three leaky ReLU variants outperform the ReLU baseline on the test set. The results indicate PReLU overfits small datasets, while RReLU is especially effective when overfitting is a concern, though its randomness requires further study.
- Overall results: All three leaky ReLU variants outperform the ReLU network baseline on the test set across CIFAR-10, CIFAR-100, and NDSB.The comparisons use ReLU as the baseline and include results from Tables 3–5.
- Leaky ReLU behavior: Normal leaky ReLU (a = 100) performs similarly to ReLU, whereas very leaky ReLU with larger a = 5.5 performs much better.
- Overfitting: On the training set, PReLU has the lowest error, while Leaky ReLU and RReLU have higher errors than ReLU, indicating possible severe PReLU overfitting on small datasets.
- Overfitting: RReLU’s superiority is more significant on NDSB, where the training set is smaller and the network is larger, supporting its effectiveness against overfitting.The paper presents this explanation as a conjecture and validation from the experiment.
- Open question: The influence of RReLU’s randomness on network training and testing remains an open question.
5. Conclusion
The paper analyzes four rectified activation functions across multiple architectures and datasets, finding that all three modified leaky ReLUs consistently outperform the original ReLU. The theoretical reasons for this advantage and performance on large-scale data remain open questions.
- The study analyzes four rectified activation functions using various network architectures on three datasets.
- All three modified leaky ReLUs consistently outperform the original ReLU.
- The theoretical reasons for the superior performance of modified leaky ReLUs lack rigorous justification.
- Performance on large-scale data remains an open question for future investigation.