Source-linked AI summary
Deep Residual Networks with Exponential Linear Unit
Anish Shah, Eashan Kadam, Hena Shah, Sameer Shinde, Sandip Shingade
TL;DR
Very deep networks face gradient-flow problems, and Residual Networks use skip connections to pass information into deeper layers. This paper replaces ReLU and Batch Normalization with ELU in Residual Networks, reporting faster learning and lower test error on CIFAR datasets, especially as depth increases.
Problem
Method
The paper uses ELU instead of the combination of ReLU and Batch Normalization in Residual Networks.
Source: §1 · p. 1, §3.1 · p. 2Results
ELU-based Residual Networks improve test error on CIFAR-10 and CIFAR-100, with the advantage increasing with depth; on CIFAR-100, test error falls from 27.23% for the original model to 26.55% for the proposed model.
Source: §1 · p. 1, §5 · p. 5, §4.2 · p. 5Takeaways & Limitations
The authors conclude that the model enables researchers to use very deep models while improving learning behavior and classification performance on CIFAR-10 and CIFAR-100.
Source: §5 · p. 6
Abstract
from arXiv · showhide
Very deep convolutional neural networks introduced new problems like vanishing gradient and degradation. The recent successful contributions towards solving these problems are Residual and Highway Networks. These networks introduce skip connections that allow the information (from the input or those learned in earlier layers) to flow more into the deeper layers. These very deep models have lead to a considerable decrease in test errors, on benchmarks like ImageNet and COCO. In this paper, we propose the use of exponential linear unit instead of the combination of ReLU and Batch Normalization in Residual Networks. We show that this not only speeds up learning in Residual Networks but also improves the accuracy as the depth increases. It improves the test error on almost all data sets, like CIFAR-10 and CIFAR-100
1. Introduction
Very deep CNNs face gradient-flow and degradation problems that hinder convergence and accuracy. The paper replaces ReLU plus Batch Normalization with ELU in Residual Networks and reports faster learning and improved accuracy on CIFAR-10 and CIFAR-100, with a larger accuracy gap as depth increases.
1. Introduction
- Very deep CNNs can suffer vanishing gradients during backpropagation, while large derivatives can also produce exploding gradients.Multiplication of small values through the chain rule can make gradients decrease exponentially and slow learning.Source: §1 · p. 1
- Normalized initialization and Batch Normalization have reduced gradient problems, while ELUs are reported to reduce vanishing gradients and improve learning speed and accuracy versus ReLU plus Batch Normalization.ELUs introduce negative values that push mean activation toward zero, reducing bias shift.Source: §1 · p. 1
- After networks begin converging, accuracy can degrade as depth increases; Highway and Residual Networks use skip connections to improve information flow into deeper layers.The 152-layer ResNet is reported to outperform other models.Source: §1 · p. 1
- ELU-based Residual Networks improve accuracy on CIFAR-10 and CIFAR-100, and their advantage over the original model grows as depth increases.The paper also reports faster learning in very deep networks.Source: §1 · p. 1
2. Background
The background distinguishes gradient-flow problems from degradation in very deep networks, and describes skip connections in Highway and Residual Networks. It also explains how ELU’s negative activations push mean activations toward zero.
2. Background
- Very deep networks face vanishing or exploding gradients that impede convergence, while adding layers can also increase training error after convergence begins.The background describes normalized initialization and Batch Normalization as approaches to gradient problems.Source: §2 · pp. 1–2
- Highway Networks use parameterized skip connections to regulate information flow into deeper layers, while ResNets use identity shortcuts without extra parameters or computation complexity.The cited ResNet account links shortcuts to improved accuracy in deeper networks.Source: §2 · p. 2
- Residual blocks combine a convolutional transformation with an identity path; the identity transformation can match dimensions and passes earlier-layer features and gradients across layers.Figure 1 illustrates the ith residual block.Source: §2 · p. 2, Figure 1 · p. 2
- ELUs include negative activations and saturate at a negative value for smaller inputs, pushing mean activations toward zero and reducing bias shift.The background connects reduced bias shift with faster learning.Source: §2 · p. 2
3. Residual Networks with Exponential Linear Unit
The section tests several ELU placements in residual blocks and settles on Conv-ELU-Conv-BN without post-addition ELU as the block used for CIFAR experiments, reporting improved learning behavior and classification performance.
3.1. ResNet Architecture
- The ResNet is described as a composition of residual blocks, with each block encoding an update rule.The example block uses Conv-BN-ReLU-Conv-BN, and the experiments reproduce that construction scheme.Source: §3.1 · p. 2
3.2. ResNet with ELU
- Replacing ReLU and Batch Normalization with ELU is the proposed change to the ResNet block design.Figure 3 illustrates the paper’s experiments with ELU placements in residual blocks.Source: §3.2 · p. 3
3.2.1 Conv-ELU-Conv-ELU
- Conv-ELU-Conv-ELU caused gradients to explode; lowering the learning rate did not make the deeper tested models converge.The 20-layer model converged only at very low accuracy, while the 56- and 110-layer models still did not converge.Source: §3.2.1 · p. 3
3.2.2 ELU-Conv-ELU-Conv
- The ELU-Conv-ELU-Conv pre-activation block also produced exploding gradients, prompting the authors to add Batch Normalization before addition.They attribute the instability to larger gradients from the exponential function and report that lowering the learning rate did not resolve it.Source: §3.2.2 · p. 3
3.2.3 Conv-ELU-Conv-BN and ELU after Addition
- Adding Batch Normalization reduced exploding gradients, but ELU after addition harmed accuracy in very deep ResNets by perturbing each skip connection.Figure 4 compares CIFAR-10 test error for the original ResNet and the post-addition ELU variant; Figure 5 compares learning behavior across 20-, 32-, 44-, 56-, and 110-layer models.Source: §3.2.3 · p. 3, Figure 4 · p. 3, Figure 5 · p. 4
3.2.4 Conv-ELU-Conv-BN and No ELU after Addition
- The Conv-ELU-Conv-BN block without ELU after addition is used in all CIFAR-10 and CIFAR-100 experiments and improves learning behavior and classification performance.The authors report that omitting ELU after addition avoids the accuracy degradation found in the preceding configuration.Source: §3.2.4 · p. 4
4. Results
The ELU-based model learns faster and improves classification performance over the original ResNet on CIFAR-10 and CIFAR-100. On CIFAR-100, its reported test error is 26.55%, compared with 27.23% for the original ResNet.
4. Results
- The experiments compare learning behavior and classification performance on CIFAR-10 and CIFAR-100, and report that our model outperformed the original ResNet on both.The study also compares classification performance with previously published state-of-the-art models.Source: §4 · p. 4
4.1. CIFAR-10 Analysis
- CIFAR-10 experiments used 32 × 32 color-normalized inputs, 6n convolutional layers, and an ELU activation immediately before global average pooling.Networks were trained with batch size 128, data augmentation, and depths of 20, 32, 44, 56, and 110 layers.Source: §4.1 · p. 4
4.1.1 Learning Behavior
- On CIFAR-10, the ELU-based model converged many epochs before the original ResNet across 20-, 32-, 44-, 56-, and 110-layer networks.The learning-rate difference increased with depth; after 125 epochs, both models converged to nearly the same value, while our model had slightly lower training loss.Source: §4.1.1 · p. 5
4.1.2 Classification Performance
- On CIFAR-10, test errors were nearly the same at 20 layers, but the ELU-based model's advantage grew as depth increased.Table 1 reports test error for each model from the epoch with the lowest validation error.Source: §4.1.2 · p. 5, Table 1 · p. 5
4.2. CIFAR-100 Analysis
- CIFAR-100 test error was 26.55% for our model versus 27.23% for the original ResNet; on CIFAR-10, the model increasingly outperformed ResNet as depth increased.The CIFAR-100 model was described as one of the best published single-model performances.Source: §4.2 · p. 5, §4.1.2 · p. 5
5. Conclusions
The paper introduces Residual Networks with ELUs, which learn faster than current Residual Networks and improve accuracy as depth increases. On CIFAR-10 and CIFAR-100, the authors report better test error and learning behavior than the current model.
5. Conclusions
- On CIFAR-10 and CIFAR-100, the ELU-based Residual Network improves beyond current state-of-the-art test error while learning faster than current models.The conclusion reports better accuracy than original Residual Networks as depth increases.Source: §5 · pp. 5–6
- Across 20-, 32-, 44-, 56-, and 110-layer models, the ELU-based model outperforms the original ResNet in classification performance.Source: Figure 6 · p. 6
- ELUs push mean activations toward zero, reducing bias shift and increasing learning speed.The paper attributes this effect to ELUs introducing small negative values.Source: §5 · p. 6