PaperlayerView exact revision on arXiv

Source-linked AI summary

Densely Connected Convolutional Networks

Gao Huang, Zhuang Liu, Laurens van der Maaten, Kilian Q. Weinberger

arXiv:1608.06993v5cs.CVcs.LG

TL;DR

Deep convolutional networks can lose information and gradients across many layers, motivating architectures with shorter paths. DenseNet connects each layer to every later layer through feature concatenation, achieving state-of-the-art results across several competitive datasets with substantially fewer parameters and less computation.

  • Problem

    Deep convolutional networks can lose input information and gradients as signals pass through many layers.

  • Method

    DenseNet connects all same-sized layers directly, concatenating feature-maps from preceding layers as inputs to each subsequent layer.

  • Results

    DenseNets achieved state-of-the-art results across several competitive datasets while requiring substantially fewer parameters and less computation.

  • Takeaways & Limitations

    DenseNets learn compact models through feature reuse and reduced feature redundancy, supporting their use as accurate convolutional feature extractors.

  • Takeaways & Limitations

    The experiments used hyperparameter settings optimized for residual networks rather than DenseNets, so more extensive tuning may improve ImageNet performance.

Abstract

from arXiv · show

Recent work has shown that convolutional networks can be substantially deeper, more accurate, and efficient to train if they contain shorter connections between layers close to the input and those close to the output. In this paper, we embrace this observation and introduce the Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion. Whereas traditional convolutional networks with L layers have L connections - one between each layer and its subsequent layer - our network has L(L+1)/2 direct connections. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. DenseNets have several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters. We evaluate our proposed architecture on four highly competitive object recognition benchmark tasks (CIFAR-10, CIFAR-100, SVHN, and ImageNet). DenseNets obtain significant improvements over the state-of-the-art on most of them, whilst requiring less computation to achieve high performance. Code and pre-trained models are available at https://github.com/liuzhuang13/DenseNet .

1. Introduction

DenseNet addresses vanishing information and gradients in increasingly deep CNNs by connecting each layer directly to all preceding layers with matching feature-map sizes. This dense connectivity improves information flow, parameter efficiency, trainability, and benchmark performance.

  • Motivation: Deep CNNs risk input information and gradients vanishing or washing out across many layers, motivating architectures that shorten signal paths.ResNets [11] and Highway Networks [34] use identity connections, while stochastic depth [13] randomly drops layers during training.
  • Dense connectivity: DenseNet connects each layer to all preceding layers with matching feature-map sizes, passing each layer’s feature-maps to every subsequent layer.This feed-forward connectivity pattern is designed to maximize information flow between layers.
  • Parameter efficiency: Dense connectivity reduces redundant feature learning, requiring fewer parameters than traditional convolutional networks.Traditional architectures pass a changing state between layers, whereas DenseNets reuse previously computed feature-maps.
  • Optimization and regularization: Direct access to the original input and loss gradients gives each layer implicit deep supervision [20], facilitating training of deeper architectures.Dense connections also exhibit a regularizing effect that reduces overfitting on smaller training tasks.
  • Empirical evaluation: Across CIFAR-10, CIFAR-100, SVHN, and ImageNet, DenseNets generally use fewer parameters at comparable accuracy and significantly outperform state-of-the-art results on most tasks.The evaluation covers four highly competitive object-recognition benchmark datasets.

2. Related Work

Prior work improves deep-network trainability through cascade layouts, bypassing paths, and increased width, while DenseNets instead emphasize feature reuse through concatenation. This design yields condensed, easy-to-train, parameter-efficient models and differs from ResNets and Inception networks.

  • Cascade architectures: Earlier cascade architectures included fully connected multilayer perceptrons trained layer-by-layer and batch-gradient-trained cascade networks, but the latter scaled only to a few hundred parameters.The passage identifies cascade structures as predecessors of the proposed dense layout.
  • Skip connections: Highway Networks [34] and ResNets [11] use bypassing paths to make very deep networks easier to optimize.Highway Networks use gating units, whereas ResNets use pure identity mappings.
  • Wide architectures: Alternative approaches increase width through Inception modules [36], wide generalized residual blocks [38], or other wide structures such as FractalNets [17].GoogLeNet’s Inception module concatenates feature-maps produced by filters of different sizes.
  • DenseNets versus prior architectures: DenseNets exploit feature reuse rather than extreme depth or width, producing condensed models that are easy to train and highly parameter-efficient.Concatenating feature-maps from different layers increases variation in subsequent inputs and improves efficiency.
  • Other innovations: Other architecture innovations include Network in Network, deeply supervised networks [20], and Ladder Networks [27] [25], which add specialized feature extraction, auxiliary supervision, or lateral connections.These methods represent additional routes to improving neural-network representations or gradient flow.

3. DenseNets

DenseNet connects every layer to all subsequent layers, concatenating preceding feature-maps to improve information flow. Its dense blocks, growth-rate control, bottleneck layers, and transition-layer compression support efficient, compact architectures.

  • Dense connectivity: DenseNet gives each layer direct access to all preceding feature-maps through concatenation, creating connections from every layer to all subsequent layers.This connectivity is intended to further improve information flow compared with ResNet skip connections, whose summation may impede information flow.
  • Pooling layers: Dense blocks separate regions with fixed feature-map sizes, while transition layers perform convolution and pooling to enable down-sampling.The architecture is divided into multiple densely connected blocks because concatenation is not viable when feature-map sizes change.
  • Growth rate: The growth rate k determines how many feature-maps each layer adds, allowing narrow layers such as k = 12 while retaining access to the block’s collective feature state.The paper reports that a relatively small growth rate is sufficient to obtain state-of-the-art results on the tested datasets.
  • Bottleneck layers: Bottleneck DenseNets insert BN-ReLU-Conv(1×1) before BN-ReLU-Conv(3×3) to reduce input feature-maps and improve computational efficiency.The bottleneck design is reported as especially effective for DenseNet.
  • Compression: DenseNet-C uses transition-layer compression with output width ⌊θm⌋, setting θ = 0.5 in the experiments to improve model compactness.Here, m is the number of feature-maps in a dense block, and θ = 1 leaves transition widths unchanged.

4. Experiments

Experiments evaluate DenseNets across CIFAR, SVHN, and ImageNet against state-of-the-art architectures, especially ResNets. DenseNets achieve strong accuracy while using parameters and computation efficiently, with particularly favorable results on CIFAR.

  • CIFAR and SVHN: 3.46% error on C10+ and 17.18% on C100+ make DenseNet-BC (L = 190, k = 40) consistently outperform the existing state-of-the-art on CIFAR.These results are significantly lower than those of wide ResNet architectures [42].
  • CIFAR and SVHN: Without compression or bottleneck layers, increasing DenseNet depth and growth rate improves performance as model capacity grows, with C10+ error decreasing from 5.24% to 4.10% to 3.74% as parameters rise from 1.0M to 7.0M to 27.2M.A similar trend is observed on C100+.
  • CIFAR and SVHN: DenseNet-BC uses parameters more efficiently than competing architectures: its 250-layer model has 15.3M parameters while consistently outperforming FractalNet and Wide ResNets with more than 30M.Bottleneck layers and dimension reduction at transition layers are identified as particularly parameter-efficient.
  • CIFAR and SVHN: On datasets without augmentation, DenseNets show pronounced improvements over prior work, including 29% and about 30% relative error reductions on C10 and C100, respectively.The reductions are from 7.33% to 5.19% on C10 and from 28.20% to 19.64% on C100.
  • ImageNet: On ImageNet, DenseNets perform on par with state-of-the-art ResNets while requiring significantly fewer parameters and less test-time computation for comparable performance.The comparison uses single-crop top-1 validation errors as functions of parameter count and FLOPs, with results shown in Figure 3.

5. Discussion

DenseNets differ from ResNets by concatenating rather than summing inputs, enabling feature reuse, compact models, and implicit deep supervision. Their dense connectivity also relates conceptually to stochastic depth while producing distinct deterministic behavior.

  • Discussion: DenseNets superficially resemble ResNets, but concatenating inputs instead of summing them leads to substantially different architectural behavior.The distinction follows directly from the differing operations in the layer inputs.
  • Model compactness: Feature concatenation lets subsequent layers access earlier feature-maps, encouraging feature reuse and producing more compact models.The experiment found weights spread across many same-block inputs, while transition layers drew across all layers in the preceding block.
  • Model compactness: Around 1/3 of the parameters suffice for DenseNet-BC to match ResNet accuracy, while 0.8M parameters achieve comparable accuracy to a 1001-layer ResNet with 10.2M parameters [12].DenseNet-BC is consistently the most parameter-efficient DenseNet variant in the reported comparison.
  • Implicit Deep Supervision: DenseNets provide implicit deep supervision because a single top classifier supervises every layer through at most two or three transition layers [20].Unlike deeply-supervised nets, DenseNets share the same loss function across layers, keeping the loss and gradient simpler.
  • Stochastic vs. deterministic connection: Dense connectivity resembles stochastic depth because randomly dropping residual layers can create direct connections between surrounding layers, although the methods differ [13].This interpretation may help explain stochastic depth’s success as a regularizer.

6. Conclusion

DenseNet connects layers with the same feature-map size directly, scales to hundreds of layers without optimization difficulties, and achieves strong accuracy with fewer parameters and less computation. Its simple connectivity rule promotes feature reuse and compact internal representations.

  • DenseNet introduces direct connections between any two layers with the same feature-map size and scales naturally to hundreds of layers without optimization difficulties.
  • DenseNets consistently improve accuracy as parameters increase, achieve state-of-the-art results across several competitive datasets, and require fewer parameters and less computation for those performances.
  • DenseNets integrate identity mappings, deep supervision, and diversified depth while enabling feature reuse and compact internal representations with reduced feature redundancy.
Loading 1608.06993v5…