Source-linked AI summary
Deep Pyramidal Residual Networks
Dongyoon Han, Jiwhan Kim, Junmo Kim
TL;DR
Residual networks concentrate feature-map growth at downsampling units, raising questions about the role of those locations. This paper introduces PyramidNet, which grows dimensions gradually and reports stronger generalization than prior architectures on CIFAR-10, CIFAR-100, and ImageNet-1k.
Problem
Residual networks concentrate feature-map growth at downsampling units, but evidence shows their importance can vary with stochastic depth and unit removal.
Method
PyramidNet gradually increases feature-map dimensions across layers and adds a novel residual unit with zero-padded identity shortcuts.
Results
PyramidNets outperform previous state-of-the-art architectures on CIFAR-10, CIFAR-100, and ImageNet-1k; ImageNet α = 300 reaches a 20.5% top-1 error rate.
Takeaways & Limitations
The reported results support gradual feature-map growth as a network-design approach, with insights the authors suggest could extend to other architectures.
Takeaways & Limitations
The evaluation excludes dropout and stochastic depth, although the authors note these regularization methods could further improve performance.
Abstract
from arXiv · showhide
Deep convolutional neural networks (DCNNs) have shown remarkable performance in image classification tasks in recent years. Generally, deep neural network architectures are stacks consisting of a large number of convolutional layers, and they perform downsampling along the spatial dimension via pooling to reduce memory usage. Concurrently, the feature map dimension (i.e., the number of channels) is sharply increased at downsampling locations, which is essential to ensure effective performance because it increases the diversity of high-level attributes. This also applies to residual networks and is very closely related to their performance. In this research, instead of sharply increasing the feature map dimension at units that perform downsampling, we gradually increase the feature map dimension at all units to involve as many locations as possible. This design, which is discussed in depth together with our new insights, has proven to be an effective means of improving generalization ability. Furthermore, we propose a novel residual unit capable of further improving the classification accuracy with our new network architecture. Experiments on benchmark CIFAR-10, CIFAR-100, and ImageNet datasets have shown that our network architecture has superior generalization ability compared to the original residual networks. Code is available at https://github.com/jhkim89/PyramidNet}
1. Introduction
The paper motivates PyramidNet by distributing feature-map growth across all layers rather than concentrating it at downsampling units, addressing their disproportionate importance in ResNets. It introduces PyramidNet and a novel residual unit, with zero-padded identity shortcuts enabling a mixture of plain and residual networks.
- Motivation: Conventional architectures increase feature-map dimension while reducing spatial size, because diversified high-level attributes benefit classification.
- Motivation: Deleting ResNet building blocks at downsampling locations, where feature-map dimension doubles, significantly increases classification error.Under stochastic depth, deleting these downsampling blocks does not degrade classification performance.
- Contributions: PyramidNet gradually increases feature-map dimensions at all residual units instead of sharply increasing them at downsampling units.The design distributes the burden associated with downsampling across all units.
- Contributions: PyramidNet uses zero-padded identity-mapping shortcuts when increasing feature-map dimension, functioning as a mixture of plain and residual networks.
- Contributions: The paper proposes a novel residual unit that can further improve the performance of ResNet-based architectures.
2. Network Architecture
PyramidNet gradually increases channel dimensions across residual units rather than only at downsampling points, using additive or multiplicative widening schemes. Its architecture also selects a specific residual building block and considers shortcut designs compatible with per-unit channel growth.
- Channel-dimension progression: PyramidNet gradually increases channel dimensions at every residual unit instead of maintaining them until downsampling.This is the architecture’s major difference from conventional networks, including original ResNet designs.
- Channel-dimension progression: In additive PyramidNet, the channel dimension increases linearly by a step factor of α/N across N residual units.The final unit of each group reaches dimension 16 + (n −1)α/3 when groups contain equal numbers of residual units.
- Channel-dimension progression: Multiplicative PyramidNet increases channel dimensions geometrically, slowly in input-side layers and sharply in output-side layers.This progression resembles the widening pattern of architectures such as VGG and ResNet.
- Residual building block: The network uses the building block illustrated in Figure 6 (d), which the authors found most promising among alternative residual-unit constructions.The building block consists of convolutional filter stacks with ReLUs and BN layers.
- Shortcut connections: Because PyramidNet increases feature-map dimensions at every unit, its shortcut options are zero-padded identity mappings or 1×1-convolution projection shortcuts.The passage notes that 1×1-convolution shortcuts produce poor results when there are too many residual units.
3. Discussions
The discussions study PyramidNet’s architecture, shortcut connections, and residual-unit design through experiments. Results indicate stronger ensemble behavior, improved generalization, and performance gains from specific residual-unit modifications.
- Architecture: Gradually increasing feature-map dimensions distributes the burden across residual units instead of doubling channels at downsampling units.This design resolves the pronounced sensitivity associated with removing downsampling units.
- Ensemble effect: PyramidNet’s ensemble effect is stronger than the original ResNet, improving generalization ability.Deleting individual units produces only slight performance loss, while downsampling-unit sensitivity is reduced.
- Shortcut connections: Because feature-map dimensions differ across units, PyramidNet uses zero-padded or projection shortcuts rather than identity mapping alone.The paper notes that projection shortcuts can hamper information propagation and cause optimization problems in very deep networks.
- Shortcut connections: Zero-padded identity-mapping shortcuts mix residual-network and plain-network behavior, with the mixture increasing as PyramidNet expands channels at every unit.This interpretation is presented as a conjecture and is supported by Figure 4’s test-error behavior.
- Residual-unit design: Removing ReLUs after residual-unit addition improves performance, while removing the first ReLU inside each block enhances performance and retaining the other preserves nonlinearity.The experiments identify the first-ReLU removal as preferable among the tested block configurations.
- Residual-unit design: Adding a batch-normalization layer at the end of each building block improves performance, including in networks that remove the first ReLU.Table 3 supports this conclusion for the tested PyramidNet and new-building-block configurations.
4. Experimental Results
Experiments on CIFAR-10, CIFAR-100, and ImageNet show that PyramidNets improve generalization over existing residual-network models. Additive PyramidNets perform especially well as model capacity increases, and the α = 300 ImageNet model achieves a 19.6% top-1 error rate.
- CIFAR experiments: PyramidNets show superior generalization on CIFAR-10 and CIFAR-100 across model parameter counts compared with state-of-the-art models.Evaluation uses top-1 error rates, with results reported for additive PyramidNets using basic and pyramidal bottleneck residual units.
- CIFAR experiments: As parameter counts increase, additive PyramidNets increasingly outperform multiplicative PyramidNets by allocating more feature-map capacity to input-side layers.Additive PyramidNets increase feature-map dimensions linearly, whereas multiplicative PyramidNets comparatively emphasize output-side layers.
- CIFAR experiments: Dropout and stochastic depth were excluded despite potentially improving PyramidNet performance, preserving a fair comparison with other models.The comparison therefore does not include these additional regularization methods.
- ImageNet experiments: ImageNet experiments use additive PyramidNets with pyramidal bottleneck residual units, deleting the first ReLU and adding batch normalization after the final layer.Models are trained for 120 epochs with scale-jittering and aspect-ratio data augmentation.
- ImageNet experiments: 19.6% top-1 error: PyramidNet with α = 300 outperforms pre-activation ResNet and Inception-ResNet-v2 on ImageNet.The model was evaluated using a 320 × 320 crop to compare with Inception-ResNet's 299 × 299 testing crop.
5. Conclusion
The paper introduces PyramidNets, which gradually increase feature-map dimensions alongside ResNets, and a novel residual unit with a zero-padded shortcut. Across CIFAR-10, CIFAR-100, and ImageNet-1k, PyramidNets outperform previous state-of-the-art deep network architectures, while future work targets principled feature-dimension optimization.
- 5. Conclusion: PyramidNets gradually increase feature-map dimensions to construct a new deep architecture alongside ResNets.The authors identify gradual feature-map expansion as the paper’s main idea.
- 5. Conclusion: The proposed residual unit adds a new building block with a zero-padded shortcut and significantly improves generalization.The supplied passage states that this design leads to significantly improved general…
- 5. Conclusion: PyramidNets outperform all previous state-of-the-art deep network architectures on CIFAR-10, CIFAR-100, and ImageNet-1k.These tests cover the three datasets named in the conclusion passage.
- 5. Conclusion: Future work will optimize feature-map dimensions and other parameters more systematically using principled cost functions for residual networks.The proposed cost functions are intended to provide insight into the nature of residual networks.