Source-linked AI summary
Comparison of non-linear activation functions for deep neural networks on MNIST classification task
Dabal Pedamonti
TL;DR
The paper evaluates alternatives to Sigmoid and studies how activation functions, depth, and weight initialization affect MNIST classification. It reports faster learning for ELU and SELU than ReLU and Leaky ReLU, while deeper networks improve performance but increase training time.
Problem
The paper addresses the need to understand the advantages and disadvantages of nonlinear activation functions, deeper networks, and weight initialization for neural-network performance.
Method
The paper compares ReLU variants on MNIST and evaluates deeper networks with multiple Gaussian and uniform weight-initialization methods.
Results
ELU performed better than Leaky ReLU and ReLU, was usually better than SELU, and ELU and SELU learned faster than ReLU and Leaky ReLU.
Takeaways & Limitations
Deeper networks can improve accuracy but require more training time, making accuracy versus computation a practical trade-off.
Takeaways & Limitations
The study did not compare all combinations of activation functions and parameters, and further error-function experiments were suggested.
Abstract
from arXiv · showhide
Activation functions play a key role in neural networks so it becomes fundamental to understand their advantages and disadvantages in order to achieve better performances. This paper will first introduce common types of non linear activation functions that are alternative to the well known sigmoid function and then evaluate their characteristics. Moreover deeper neural networks will be analysed because they positively influence the final performances compared to shallower networks. They also strictly depend on the weight initialisation hence the effect of drawing weights from Gaussian and uniform distribution will be analysed making particular attention on how the number of incoming and outgoing connection to a node influence the whole network.
1. Introduction
The paper compares ReLU variants on MNIST and examines how deeper architectures and weight initialization affect network performance.
- ReLU replaced Sigmoid as a widely used activation because of its positive impact on machine-learning tasks.
- The experiment evaluates ReLU variants on MNIST, which contains 50,000 training, 10,000 validation, and 10,000 test images of 10 digits.
- Networks up to the 8th hidden layer are compared with shallower structures using accuracy and error.
- Weight initialization is examined through uniform methods based on fan_in and fan_out variance assumptions.
- The first experiment uses batch size 50 for 100 epochs, while the second uses the same batch size for 50 epochs.
2. Activation functions
The paper presents ReLU variants designed to address gradient and bias-shift issues in deeper networks, including Leaky ReLU, ELU, and SELU.
- Gradient behavior: The activation variants aim to address vanishing and exploding gradients that can arise when derivative terms are repeatedly multiplied in deeper architectures.
- Leaky ReLU: Leaky ReLU adds a nonzero negative-side gradient to reduce dying units and make optimization more robust.
- ELU: ELU uses an exponential negative branch and pushes mean activations toward zero to reduce bias shift.
- ELU: ELU is described as faster-learning than ReLU because centering activations at zero reduces bias shift without batch normalization.
- SELU: SELU uses α = 1.6733 and λ = 1.0507 and is described as self-normalising across many layers.
3. Experimental comparison of activation functions
The experiment compares ReLU variants on MNIST across learning rates, then selects ELU for deeper-network analysis. ELU generally outperforms ReLU, Leaky ReLU, and SELU under the reported comparisons, while learning-rate choice balances loss, accuracy, and generalization.
- Experimental setup: The study evaluates Leaky ReLU, ELU, and SELU with two hidden layers of 100 units, comparing them against sigmoid and ReLU baselines.The experiments use MNIST and vary the learning rate across 0.01, 0.05, 0.1, and 0.2.
- Learning-rate comparison: Lower learning rates produce smaller losses, whereas higher learning rates improve accuracy, requiring a trade-off between the two metrics.Learning rates 0.05 and 0.1 gave the best combined results, and validation accuracy increased by 2–4% when increasing the rate from 0.05 to 0.1.
- Learning-rate comparison: High learning rates cause earlier validation overfitting and generalize less despite sometimes achieving higher accuracy.The ELU error and accuracy are examined as functions of learning rate in Figures 1 and 2.
- Activation-function comparison: ELU was selected for deeper-network experiments because it generally performed better than Leaky ReLU, ReLU, and SELU across loss, accuracy, and generalization.SELU occasionally performed better, especially at learning rate 0.05, but the reported selection followed multiple runs.
- Activation-function comparison: ELU achieved 2% higher accuracy than SELU, with consistently lower loss in the reported comparison.Both functions began overfitting around epoch 20, although ELU started slightly later.
- Activation-function comparison: ReLU and its variants consistently achieved higher accuracies than sigmoid in the reported MNIST comparisons.This result is presented as consistent with the prior motivation for examining ReLU-based activations.
4. Deep neural network experiments
Deeper ELU and SELU networks generally improved validation performance, while results depended on weight initialization and deeper models required greater computation. ELU favored Glorot uniform initialization, whereas SELU favored Gaussian initialization over uniform.
- ELU experiments: Fan_out produced accuracies similar to Glorot uniform but higher average losses, despite very small training-set losses.The comparison distinguishes accuracy behavior from loss behavior across initializers.
- ELU experiments: Validation accuracy increased with depth for all four ELU weight-initialization methods, with Glorot uniform starting at 0.980 versus 0.978 for the others at two hidden layers.The comparison covered Glorot uniform, fan_in, fan_out, and Glorot Gaussian initialization.
- SELU experiments: For SELU, accuracy increased with depth under both initializations, but Gaussian weights produced higher accuracy and more stable errors than Glorot uniform weights.The Gaussian-versus-uniform comparison used SELU and was highlighted for a seven-hidden-layer model.
- Interpretation: Training accuracy reached 1 for all weight-initialization methods, while validation accuracy and loss changed with initialization and depth.The study focused on validation behavior because training results were uniformly saturated.
5. Conclusions
The paper compares ReLU variants, deeper networks, and weight-initialization strategies on MNIST. It reports faster learning for ELU and SELU than for ReLU and Leaky ReLU, while identifying further error-function comparisons as an open direction.
- Contributions: The study evaluates ReLU, Leaky ReLU, ELU, SELU, deeper networks, and several weight-initialization methods.The initialization methods include Glorot uniform, fan_in, fan_out, Glorot Gaussian, and SELU Gaussian.
- Findings: ELU and SELU learn faster than ReLU and Leaky ReLU in the reported experiments.The conclusion presents this as consistent with cited prior work on ELU and SELU.
- Future work: The paper leaves investigation of different error functions and their relationship to the activation-function findings for future work.This is the stated future research direction.