Source-linked AI summary
Resnet in Resnet: Generalizing Residual Architectures
Sasha Targ, Diogo Almeida, Kevin Lyman
TL;DR
ResNets provide strong optimization benefits through identity shortcuts, but their fixed residual structure makes information removal and deeper residual transformations difficult. The paper introduces RiR, a dual-stream architecture that combines residual and transient processing within generalized residual blocks. RiR outperforms the original ResNet and achieves state-of-the-art results on CIFAR-100.
Problem
Existing ResNets have difficulty learning information removal and constrain residual learning to shallow subnetworks.
Method
RiR uses generalized residual blocks with parallel residual and transient streams connected by same-stream and cross-stream transformations.
Results
RiR outperforms the original ResNet and achieves state-of-the-art results on CIFAR-100.
Takeaways & Limitations
Generalized residual blocks retain identity-shortcut optimization while supporting standard CNN, ResNet, and intermediate processing patterns.
Abstract
from arXiv · showhide
Residual networks (ResNets) have recently achieved state-of-the-art on challenging computer vision tasks. We introduce Resnet in Resnet (RiR): a deep dual-stream architecture that generalizes ResNets and standard CNNs and is easily implemented with no computational overhead. RiR consistently improves performance over ResNets, outperforms architectures with similar amounts of augmentation on CIFAR-10, and establishes a new state-of-the-art on CIFAR-100.
1 INTRODUCTION
ResNets improve optimization through identity shortcuts but have limitations in learning information removal and deep residual transformations. RiR addresses these issues with parallel residual and non-residual streams, achieving state-of-the-art CIFAR-100 performance.
- ResNets use ungated identity shortcuts to transmit untransformed information across layers and support optimization of extremely deep networks.The paper notes ResNets achieved state-of-the-art ILSVRC 2015 performance and enabled networks exceeding 1000 layers.
- Learning the additive inverse of identity weights is difficult, limiting a residual network’s ability to remove information from representations.
- Fixed-size residual blocks restrict residual learning to shallow subnetworks despite evidence that deeper networks are more expressive.
- RiR combines parallel residual and non-residual streams to retain shortcut-based optimization while improving expressivity and information removal.The paper reports state-of-the-art performance on CIFAR-100.
2 GENERALIZING RESIDUAL NETWORK ARCHITECTURES
Generalized residual blocks combine residual and transient streams with cross-stream transformations and shortcuts. Repeated blocks can represent standard CNNs, ResNet blocks, and intermediate architectures, motivating RiR.
- A generalized residual block contains parallel residual and transient streams, with identity shortcuts in the residual stream and standard convolution in the transient stream.The block includes same-stream transformations and cross-stream filters transferring information between streams.
- Same-stream and cross-stream activations are summed, with the residual shortcut, before batch normalization and ReLU produce the output states.The operation is summarized by Equation 1.
- The transient stream processes information nonlinearly without shortcuts, allowing information from earlier states to be discarded.
- By zeroing either stream, the generalized block can behave as a standard CNN layer or a single-layer ResNet block.
- Repeating generalized blocks provides variable-depth residual processing and can represent intermediate architectures, including the standard 2-layer ResNet block.Replacing each convolutional layer in a ResNet block with a generalized block yields RiR.
3 EXPERIMENTS
Experiments on CIFAR-10 and CIFAR-100 compare the proposed architectures with CNNs, ResNets, and other state-of-the-art systems. RiR consistently outperforms the original ResNet and reaches state-of-the-art CIFAR-100 performance.
- The evaluation selects results after grid searches over learning rate, regularization, initialization, optimizer, and shortcut type.The reported training setup includes SGD with momentum 0.9, minibatches of 500, L2 penalty 0.0001, and 82 epochs.
- RiR outperforms the original ResNet, while ResNet Init consistently improves over standard CNN architectures.
- RiR achieves competitive CIFAR-10 results using only standard random-crop and horizontal-flip augmentation.
- RiR achieves state-of-the-art results on CIFAR-100.
4 RELATED WORK
The generalized residual block relates to gated and recurrent architectures through interacting transformation streams, but differs by using ungated information flow and no additional parameters over a standard feedforward network.
- Unlike highway networks, generalized residual blocks do not use input-dependent carry and transform gates for information flow.
- Unlike LSTM and Grid-LSTM states, the residual and transient streams can be implemented without additional parameters over a standard feedforward network.
5 CONCLUSION
The paper presents a generalized residual architecture, implemented through ResNet Init and instantiated as RiR, achieving state-of-the-art results.
- The generalized residual architecture combines residual and standard convolutional networks through a modified initialization scheme called ResNet Init.
- Applying ResNet Init to the original ResNet produces the novel RiR architecture.
- RiR achieves state-of-the-art results.
6 APPENDIX
The appendix describes the relationships among CNN, ResNet, ResNet Init, and RiR architectures, their experimental configurations, and the single-layer implementation of generalized residual blocks.
- Figure 2 relates standard CNN, ResNet, ResNet Init, and RiR architectures.
- Figure 3 examines the effect of ablating each stream in the generalized residual network architecture.
- The experiments include baseline 32-layer CNNs, 18-layer and wide CNN architectures, and RiR models with varied blocks and layers per block.
- 6.1 GENERALIZED RESIDUAL BLOCK IMPLEMENTATION: ResNet Init is implemented by combining the identity shortcut and linear transformation in a single convolutional or fully connected layer after concatenating residual and transient streams.
- 6.1 GENERALIZED RESIDUAL BLOCK IMPLEMENTATION: The single-layer implementation is exactly equivalent to separate linear operations before accounting for weight regularization.Under L2-style regularization, equivalence requires subtracting the partial identity from the weights before weight decay.