PaperlayerView exact revision on arXiv

Source-linked AI summary

Parametric Exponential Linear Unit for Deep Convolutional Neural Networks

Ludovic Trottier, Philippe Giguère, Brahim Chaib-draa

arXiv:1605.09332v4cs.LGcs.CVcs.NE

TL;DR

Object recognition requires effective visual representations, while ELU’s parameter must be set by hand. The paper proposes learning ELU’s parameterization as PELU and reports better performance than ELU across evaluated benchmarks, including a 7.29% relative ImageNet error improvement for NiN with 24 additional parameters.

  • Problem

    ELU helps manage bias shift in CNNs but requires a parameter to be set by hand, and suitable parameterizations can vary across networks.

  • Method

    PELU learns ELU parameterization during back-propagation, controlling activation shape while preserving differentiability and using 2L additional parameters.

  • Results

    PELU outperformed ELU across the reported MNIST, CIFAR-10/100, and ImageNet evaluations, including a 7.29% relative TOP-1 error improvement for NiN on ImageNet.

  • Takeaways & Limitations

    Vgg used PELU’s flexibility to learn different activation shapes at different network locations.

Abstract

from arXiv · show

Object recognition is an important task for improving the ability of visual systems to perform complex scene understanding. Recently, the Exponential Linear Unit (ELU) has been proposed as a key component for managing bias shift in Convolutional Neural Networks (CNNs), but defines a parameter that must be set by hand. In this paper, we propose learning a parameterization of ELU in order to learn the proper activation shape at each layer in the CNNs. Our results on the MNIST, CIFAR-10/100 and ImageNet datasets using the NiN, Overfeat, All-CNN and ResNet networks indicate that our proposed Parametric ELU (PELU) has better performances than the non-parametric ELU. We have observed as much as a 7.28% relative error improvement on ImageNet with the NiN network, with only 0.0003% parameter increase. Our visual examination of the non-linear behaviors adopted by Vgg using PELU shows that the network took advantage of the added flexibility by learning different activations at different layers.

I. INTRODUCTION

Object recognition supports complex scene understanding, while CNN activation functions affect feature learning and bias shift. PELU addresses ELU’s hand-set parameter through learned, differentiable parameterization and is evaluated across datasets and architectures.

  • Motivation: Object recognition enables visual systems to identify scene elements and build high-level representations for tasks including face verification, robotic grasping, and autonomous driving.
  • Motivation: CNNs transform high-dimensional observations into increasingly abstract concepts through hierarchical nonlinear feature compositions.
  • Activation functions: ELU can reduce bias shift, which otherwise may cause learning oscillations and impede optimization.
  • Contributions: PELU learns ELU parameterization during back-propagation, preserving differentiability while adding only 2L parameters and ELU-level computational complexity.
  • Contributions: The paper evaluates PELU on MNIST, CIFAR-10/100, and ImageNet with ResNet, Network in Network, All-CNN, Vgg, and Overfeat architectures.
  • Related work: Unlike Maxout, PELU avoids multiplying layer weights by K and remains computationally comparable to ELU.

III. PARAMETRIC EXPONENTIAL LINEAR UNIT

PELU extends ELU with learnable parameters that control its saturation, decay, and positive slope while enforcing differentiability and positivity during training.

  • ELU limitation: ELU is identity for positive inputs and a(exp(h)−1) for negative inputs, with a conventionally fixed to preserve differentiability at h = 0.
  • Parameterization: PELU introduces positive parameters a, b, and c to control saturation, exponential decay, and the positive-side linear slope.
  • Parameterization: The original ELU is recovered when a = b = c = 1.
  • Parameter effects: Increasing a lowers the negative saturation point, increasing b reduces exponential decay, and increasing c steepens the positive linear slope.
  • Differentiability: PELU enforces differentiability at h = 0 by equating derivatives on both sides and solving c = a/b.
  • Training: PELU is trained jointly with network parameters through back-propagation, while a and b are constrained to remain greater than 0.1.

IV. EXPERIMENTATIONS

The experiments evaluate supervised learning with PELU on CIFAR-10/100 and ImageNet tasks.

  • Experimental scope: The experiments use supervised learning on CIFAR-10/100 and ImageNet tasks.
  • Experimental scope: CIFAR-10/100 are included as object-recognition benchmarks.
  • Experimental scope: ImageNet is included as an additional experimental task.

A. MNIST Auto-Encoder

The DAA-net auto-encoder evaluates PELU for unsupervised feature learning on unlabeled MNIST images, comparing reconstruction performance with ELU and ReLU references.

  • A. MNIST Auto-Encoder: The DAA-net is a deep auto-encoder trained on unlabeled MNIST images for learning feature representations.It uses four fully connected encoder layers of sizes 1000, 500, 250, and 30, with a symmetrical untied decoder.
  • A. MNIST Auto-Encoder: The MNIST comparison includes PELU, ELU, and BN-ReLU as reference activations.
  • A. MNIST Auto-Encoder: PELU outperformed ELU and ReLU in both convergence speed and reconstruction error.The comparison uses test mean squared error averaged over five training tries.
  • A. MNIST Auto-Encoder: 75 epochs and 1.04e−4 test MSE were reported for PELU, versus 100 epochs and 1.12e−4 for ELU.ReLU also converged at epoch 100 with a test MSE of 1.49e−4.

B. CIFAR-10/100 Object Recognition

A 110-layer ResNet was evaluated on CIFAR-10 and CIFAR-100 with PELU, ELU, and additional normalized activation references. PELU improved error and convergence relative to ELU with minimal parameter overhead, while BN before activation worsened performance.

  • B. CIFAR-10/100 Object Recognition: The CIFAR experiments used a 110-layer residual network with activation replacement as the main comparison change.The network used basic and transition blocks, with transition blocks combining spatial subsampling and zero concatenation for CIFAR.
  • B. CIFAR-10/100 Object Recognition: PELU achieved lower minimum median test errors than ELU on CIFAR-10 and CIFAR-100: 5.36% versus 5.99%, and 24.55% versus 25.08%.
  • B. CIFAR-10/100 Object Recognition: 10.52% and 2.11% relative error improvements over ELU were reported for PELU on CIFAR-10 and CIFAR-100, respectively.
  • B. CIFAR-10/100 Object Recognition: PELU showed steadier convergence than ELU, avoiding the same magnitude of late-stage test-error increases.ELU exhibited large error increases at the end of the second training stage and a smaller increase at training completion.
  • B. CIFAR-10/100 Object Recognition: BN before ELU or PELU worsened performance, although batch normalization remained after the second convolutional layer.

C. Understanding the effect of Batch Normalization

The study examines batch normalization placed before ELU or PELU in a 110-layer ResNet on CIFAR-10 and CIFAR-100. In both cases, this placement increased error, with a larger relative increase for ELU.

  • C. Understanding the effect of Batch Normalization: Adding BN before ELU or PELU caused large test-error increases on both CIFAR datasets.
  • C. Understanding the effect of Batch Normalization: 73% and 39% relative error increases occurred for ELU on CIFAR-10 and CIFAR-100, compared with 9% and 3% for PELU.
  • C. Understanding the effect of Batch Normalization: With BN before activation, ELU errors rose to 10.39% on CIFAR-10 and 34.75% on CIFAR-100, while PELU errors rose to 5.85% and 25.38%.
  • C. Understanding the effect of Batch Normalization: The authors conclude that PELU should not be preceded by BN, despite its smaller degradation than ELU.BN was still used after the second convolutional layer in the residual block.

D. ImageNet Object Recognition

The paper evaluates PELU against ELU and BN+ReLU across four ImageNet network architectures under two training regimes. PELU achieves the lowest error rates, with performance depending on the training regime.

  • PELU, ELU, or BN+ReLU replaced the activation module in ResNet18, NiN, All-CNN, and Overfeat ImageNet experiments.
  • 36.06% TOP-1 error was achieved by NiN with PELU versus 40.40% with ELU, a 7.29% relative improvement.Only 24 additional parameters were added to the network.
  • Regime #1 produced a larger ELU–PELU performance gap than regime #2.Regime #1 used a higher initial learning rate and larger learning-rate decay.
  • PELU had the lowest TOP-1 error rates for all four networks on the ImageNet 2012 validation set.
  • All-CNN and Overfeat showed a small PELU error-rate increase from epoch 44, while ELU and ReLU remained steady.The authors suggest larger learning rates and decays help PELU improve performance.

E. Experimenting with Parameter Configuration

The paper compares four PELU parameter configurations in a 110-layer ResNet on CIFAR-10 and CIFAR-100. The (a, 1/b) configuration performs best, while configurations using 1/b retain lower error than those using b.

  • The four evaluated PELU configurations are (a, b), (a, 1/b), (1/a, b), and (1/a, 1/b).Weight decay makes these configurations non-reciprocal in practice.
  • The (a, 1/b) configuration obtained the best accuracy in the CIFAR-10 and CIFAR-100 experiments.
  • 5.36% and 24.55% minimum test error medians were obtained by (a, 1/b) on CIFAR-10 and CIFAR-100, respectively.The compared configurations had higher reported error medians on both datasets.
  • The two configurations using 1/b had significantly lower error rates than the two configurations using b.Configurations using b also showed larger error increases during the second training stage.
  • Using 1/b helps the activation retain its nonlinear nature as b decreases, unlike parameterizations using b.

F. Parameter Progression

PELU parameters adopted different trajectories across Vgg layers during CIFAR-10 training. Several layers converged to ReLU-like activations, while most retained nonzero negative saturation.

  • Layers 2, 4, 7, and 10 converged to slope near 1 and negative saturation near 0, indicating ReLU-shaped activations.The authors connect this shape with ReLU’s activation sparsity effect.
  • At several layers, the slope first increased sharply, then decreased and converged near 1.The authors believe the initial increase may help disentangle redundant neurons, but state that the reason is not fully understood.
  • Vgg learned different nonlinear behaviors throughout training across its layers.Figure 8 tracks slope and negative saturation for each layer.
  • Layer 1’s negative saturation converged near 0.5, while layer 14’s converged near 2.These nonzero values mean the learned activations output negative values for negative arguments.
  • Most layers retained nonzero negative saturation even though PELU could learn zero negative saturation.The authors interpret this as additional evidence that negative outputs help manage bias shift.

V. DISCUSSION

The experiments omitted Batch Normalization before ELU and PELU because the authors observed a detrimental effect in their ResNet CIFAR experiments.

  • Batch Normalization was not used before ELU or PELU in the experiments.The authors observed that preceding these activations with BN increased error rates in ResNet experiments on CIFAR-10 and CIFAR-100.
  • The authors report that BN before ELU and PELU increased error rates but reduced error rates before ReLU.They state that the reason for this differing effect is unclear.

VI. CONCLUSION

PELU learns the ELU activation parameterization instead of requiring it to be set by hand, and experiments report better CNN performance than ELU. Vgg learned different activation shapes at different network locations using PELU.

  • PELU learns the ELU activation parameterization, alleviating ELU’s requirement for a hand-set parameter.
  • Experiments on CIFAR-10/100 and ImageNet with ResNet, NiN, All-CNN, and Overfeat report better performance for PELU than ELU.
  • Vgg learned different activation shapes at different locations in the network, using PELU’s added flexibility.
Loading 1605.09332v4…