Source-linked AI summary
ResNet or DenseNet? Introducing Dense Shortcuts to ResNet
Chaoning Zhang, Philipp Benz, Dawit Mureja Argaw, Seokju Lee, Junsik Kim, Francois Rameau, Jean-Charles Bazin, In So Kweon
TL;DR
ResNet’s identity shortcuts stabilize training but constrain representation capacity, while DenseNet’s feature concatenation raises GPU-memory and computation costs. The paper unifies their analysis through dense summation and proposes dense weighted normalized shortcuts. Across benchmark evaluations, DSNet outperforms ResNet and matches or slightly exceeds DenseNet with fewer computation resources.
Problem
ResNet limits representation capacity through identity shortcuts, whereas DenseNet’s dense concatenation requires substantial GPU memory and computation, creating a performance–resource dilemma.
Method
The paper analyzes ResNet and DenseNet through a unified dense-summation perspective and replaces identity shortcuts with dense weighted normalized shortcuts.
Results
DSNet outperforms ResNet by a significant margin and achieves comparable or slightly better performance than DenseNet with fewer parameters and computation resources.
Takeaways & Limitations
Dense shortcuts address ResNet’s representational-capacity decrease while avoiding DenseNet’s greater GPU-resource requirements.
Takeaways & Limitations
The unified dense-summation analysis establishes a theoretical connection, while practical GPU implementations have different efficiency characteristics.
Abstract
from arXiv · showhide
ResNet or DenseNet? Nowadays, most deep learning based approaches are implemented with seminal backbone networks, among them the two arguably most famous ones are ResNet and DenseNet. Despite their competitive performance and overwhelming popularity, inherent drawbacks exist for both of them. For ResNet, the identity shortcut that stabilizes training also limits its representation capacity, while DenseNet has a higher capacity with multi-layer feature concatenation. However, the dense concatenation causes a new problem of requiring high GPU memory and more training time. Partially due to this, it is not a trivial choice between ResNet and DenseNet. This paper provides a unified perspective of dense summation to analyze them, which facilitates a better understanding of their core difference. We further propose dense weighted normalized shortcuts as a solution to the dilemma between them. Our proposed dense shortcut inherits the design philosophy of simple design in ResNet and DenseNet. On several benchmark datasets, the experimental results show that the proposed DSNet achieves significantly better results than ResNet, and achieves comparable performance as DenseNet but requiring fewer computation resources.
1. Introduction
ResNet and DenseNet offer complementary strengths but expose a performance–resource dilemma: identity shortcuts limit representation capacity, while dense concatenation increases memory and computation demands. The paper addresses this dilemma with dense normalized shortcuts and reports stronger results than ResNet with fewer resources than DenseNet.
- ResNet’s identity shortcut preserves preceding features but limits representation power and learning capacity.
- DenseNet’s dense concatenation preserves preceding-layer features and can outperform ResNet with fewer parameters, but requires heavy GPU memory.
- The performance–GPU-resource trade-off makes choosing between ResNet and DenseNet nontrivial for broad applications.
- The paper proposes dense normalized shortcuts that replace ResNet’s identity shortcut while retaining its convolutional-block backbone.
- DSNet outperforms ResNet by a significant margin and achieves comparable or slightly better performance than DenseNet with fewer parameters and computation resources.
2. Related works
Related work frames ResNet and DenseNet as influential backbone architectures within broader network-design research. Their central distinction is between identity shortcuts, which stabilize training, and dense concatenation, which connects features across layers.
- Deep CNN design research includes both micro-module techniques and macro-architecture design for improving computer-vision performance.
- Normalization, attention, group convolution, and bottleneck design are examples of micro-modules inserted into existing macro-architectures.
- ResNet and DenseNet became favored backbone choices because they combine competitive performance with relatively simple designs.
- ResNet variants such as WideResNet and ResNeXt retain identity shortcuts, whereas DenseNet uses concatenation without identity shortcuts.
3. Proposed approach
The paper reframes ResNet and DenseNet as sharing dense summation while differing in how preceding feature maps are weighted or combined. It then proposes dense weighted normalized shortcuts, yielding DSNet as a ResNet-backbone alternative to identity shortcuts and dense concatenation.
- Unified perspective: ResNet and DenseNet both connect to preceding feature maps, but ResNet uses summation whereas DenseNet uses concatenation.This unified view identifies their connection pattern as shared and their practical distinction as the combination operation.
- Unified perspective: Dense concatenation before convolution is theoretically equivalent to dense summation after convolution.The equivalence provides the basis for analyzing both architectures through dense summation.
- Unified perspective: ResNet shares one convolution weight across preceding outputs, whereas DenseNet uses different weights, giving DenseNet more feature-use flexibility but increasing memory and training-time demands.DenseNet’s larger effective input-channel count contributes to higher GPU-memory use and longer training.
- Dense shortcut and DSNet: DSNet replaces ResNet’s identity shortcut with a dense weighted normalized shortcut while retaining the ResNet convolutional-block backbone.The main variant connects densely to aggregation outputs Y_l rather than only individual block outputs X_l.
- Ablation study: The naive dense identity shortcut does not improve performance over original ResNet50, whereas experiments support using aggregation outputs and show DS2Net outperforming DSNet.The ablation study also reports that DSNet-a is inferior to DSNet.
- Dense shortcut and DSNet: The proposed dense shortcut combines normalization with channel-wise weighting so preceding feature maps are similarly scaled and can receive learned importance.Normalization is intended to prevent one feature from dominating the summation, while learned weights provide flexibility.
4. Experimental Results and Analysis
Experiments across CIFAR, ImageNet, COCO, and Grad-CAM evaluations show that dense normalized shortcuts consistently improve or match established backbones while reducing resource demands in key comparisons.
- CIFAR experiments: DSNet50 (×0.25) outperforms ResNet50 (×0.25) by 2% on CIFAR-100, while DS2Net improves further.
- CIFAR experiments: DSNet consistently outperforms ResNet across a wide range of depths and widths, and also improves ResNeXt with a significant margin.
- CIFAR experiments: Dense weight normalized shortcuts outperform networks without dense connections and surpass ResNet variants and DenseNets on CIFAR-100.
- ImageNet experiments: DS2Net50 performs significantly better than ResNet50 on ImageNet and achieves equivalent or better performance than much deeper or more resource-intensive alternatives.
- ImageNet experiments: The proposed dense weighted normalized shortcut is beneficial to convergence, with DS2Net showing much smaller training error.
- COCO and visualization: DSNet achieves better performance than ResNet50 on COCO detection, while Grad-CAM shows sharper object-focused attention than ResNet50.
- Implementation and efficiency: Normalization is shared across dense shortcuts to avoid unnecessary computation and parameter overhead, and DS2Net performs relatively better than ResNet152 and DenseNet264 in memory and speed.
5. Conclusions
The paper unifies ResNet and DenseNet through dense summation and proposes dense weighted normalized shortcuts to combine representational capacity with lower resource demands.
- The core difference between ResNet and DenseNet is whether convolution parameters are shared for preceding feature maps.
- Dense weighted normalized shortcuts outperform identity shortcuts and dense concatenation while avoiding reduced representational capacity and increased GPU requirements.
- Across benchmark datasets, DSNet outperforms ResNet and achieves comparable performance to DenseNet with fewer parameters or computation.