PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual Path Networks

Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, Jiashi Feng

arXiv:1707.01629v2cs.CV

TL;DR

Residual and densely connected networks offer complementary path behaviors, but their advantages and limitations require a unified architecture. The paper proposes DPN, combining feature re-usage with new-feature exploration, and reports strong performance across classification, detection, and segmentation.

  • Problem

    Residual and densely connected networks use different path topologies, motivating analysis of their respective advantages and limitations for representation learning.

  • Method

    DPN combines residual paths for common-feature re-usage with densely connected paths for new-feature exploration.

  • Results

    DPN achieves superior performance across image classification, object detection, and semantic segmentation, with reported gains in accuracy and efficiency.

  • Takeaways & Limitations

    The dual path architecture is presented as broadly applicable while retaining high accuracy, parameter efficiency, low computational cost, and low GPU memory consumption.

  • Takeaways & Limitations

    The DPN hyper-parameters were chosen from prior experience rather than grid-search experiments because of limited computational resources.

Abstract

from arXiv · show

In this work, we present a simple, highly efficient and modularized Dual Path Network (DPN) for image classification which presents a new topology of connection paths internally. By revealing the equivalence of the state-of-the-art Residual Network (ResNet) and Densely Convolutional Network (DenseNet) within the HORNN framework, we find that ResNet enables feature re-usage while DenseNet enables new features exploration which are both important for learning good representations. To enjoy the benefits from both path topologies, our proposed Dual Path Network shares common features while maintaining the flexibility to explore new features through dual path architectures. Extensive experiments on three benchmark datasets, ImagNet-1k, Places365 and PASCAL VOC, clearly demonstrate superior performance of the proposed DPN over state-of-the-arts. In particular, on the ImagNet-1k dataset, a shallow DPN surpasses the best ResNeXt-101(64x4d) with 26% smaller model size, 25% less computational cost and 8% lower memory consumption, and a deeper DPN (DPN-131) further pushes the state-of-the-art single model performance with about 2 times faster training speed. Experiments on the Places365 large-scale scene dataset, PASCAL VOC detection dataset, and PASCAL VOC segmentation dataset also demonstrate its consistently better performance than DenseNet, ResNet and the latest ResNeXt model over various applications.

1 Introduction

The paper analyzes residual and densely connected path topologies and proposes DPN to combine feature re-usage with new-feature exploration. Experiments report high accuracy alongside efficiency and broad task performance.

  • Motivation: Skip connections provide direct information paths and facilitate gradient back-propagation, easing optimization and mitigating gradient vanishing.They let upper layers access distant lower-layer information during forward propagation.
  • Existing path topologies: ResNet adds input features to micro-block outputs, whereas DenseNet concatenates inputs with outputs from all previous micro-blocks.DenseNet’s densely connected width grows linearly with depth, increasing parameters quadratically while improving parameter efficiency.
  • Unified view: The unified HORNN view identifies residual networks with shared-layer connections and densely connected networks with shared-step weights.This analysis associates residual paths with feature re-usage and densely connected paths with new-feature exploration.
  • Proposed architecture: DPN combines residual and densely connected paths to support feature re-usage and new-feature exploration.The architecture is presented as more parameter-efficient, computationally cheaper, and less memory-intensive than state-of-the-art classification networks.
  • Evaluation: Experiments report superior DPN accuracy on ImageNet-1k and Places365, with additional gains on object detection and semantic segmentation.The paper presents the architecture as broadly applicable across these tasks.

2 Related work

Related work develops deep architectures through network design and analysis of skip connections. This paper extends that line by interpreting DenseNet through HORNN, viewing ResNet as a special case, and proposing DPN.

  • Architecture design: Advanced neural network architectures improve image classification and can directly benefit other vision tasks.AlexNet, VGG, and ResNet established increasingly deep convolutional designs and skip connections as influential approaches.
  • Architecture analysis: Prior work analyzed residual paths, connected residual networks with RNNs, and unified several residual functions.These studies sought to explain and improve established architectures rather than only introducing new ones.
  • Paper contribution: This paper interprets densely connected networks through HORNN and explains residual networks as a special case of densely connected networks.It then uses this analysis to motivate the Dual Path Network.
  • Topology overview: Figure 1 presents topological relations among residual networks, RNNs, densely connected networks, and HORNNs.Its labels identify z−1 as a time-delay unit, ⊕ as element-wise summation, and I(·) as identity mapping.

3 Revisiting ResNet, DenseNet and Higher Order RNN

The paper reframes DenseNet and ResNet within the HORNN framework, showing that their connection topologies differ in parameter sharing and feature behavior. This analysis motivates a dual-path design that combines feature exploration with feature reuse.

  • HORNN view: The analysis concludes that both residual and densely connected networks can be represented as HORNNs under appropriate parameter-sharing conditions.This equivalence provides the conceptual basis for developing the dual path network architecture.
  • HORNN view: The generalized HORNN formulation describes densely connected networks as using distinct parameters at each micro-block, enabling new information extraction from previous states.For DenseNet, neither the feature-extracting functions nor the transformation functions are shared across steps.
  • HORNN view: Residual networks are special cases of densely connected networks when the feature-extracting functions are shared across steps.The paper introduces an intermediate state to rewrite the generalized update rule into a residual-network form.
  • Topology trade-offs: Shared functions in residual networks encourage feature reuse and reduce redundancy but make exploring new features difficult.The information-sharing strategy repeatedly exposes the same feature from a given output state.
  • Topology trade-offs: Unshared functions in densely connected networks support new-feature exploration but may repeatedly extract the same feature type, increasing redundancy.The paper identifies this redundancy as an inherent limitation of the densely connected topology.

4 Dual Path Networks

DPN combines residual-like feature reuse with densely connected new-feature exploration in a modular dual path architecture. Its residual backbone and thin dense path are designed for efficient implementation and reduced complexity.

  • Dual Path Architecture: The dual path architecture shares feature-extracting functions across blocks while retaining a densely connected path for flexible new-feature learning.This design aims to reduce redundancy without removing the dense path's exploratory capacity.
  • Dual Path Architecture: DPN combines a residual path for common-feature reuse with a densely connected path for exploring new features.The two paths are integrated before the final transformation function generates the current state.
  • Dual Path Architecture: DPN is a modular convolutional-network family whose micro-blocks can be customized for task-specific use or further performance improvements.The architecture contains both residual-like and densely connected-like paths.
  • Network Structure: DPN micro-blocks use bottleneck convolutions, then split outputs between element-wise residual addition and dense-path concatenation.The bottleneck sequence is 1 × 1, 3 × 3, and 1 × 1 convolution, followed by the two-way output split.
  • Network Structure: The residual backbone plus thin dense path slows dense-path width growth and makes DPN easy to implement on existing ResNet or ResNeXt networks.The paper describes implementation through an added slice layer and concat layer, with no extra computational or memory cost under a well-optimized platform.
  • Complexity Analysis: DPNs were intentionally designed with smaller model sizes and fewer FLOPs than state-of-the-art ResNeXts, using hyperparameters chosen without grid search because of limited computational resources.The reported design settings were based on prior experience.
  • Complexity Analysis: 26% fewer parameters and 25% fewer FLOPs are reported for DPN-98 than ResNeXt-101 (64 × 4d).The FLOPs comparison uses 224 × 224 inputs and multiply-add operations; actual runtime can also depend on GPU bandwidth and coding quality.

5 Experiments

Experiments evaluate DPNs across image classification, scene classification, object detection, and semantic segmentation, comparing accuracy, efficiency, and training cost with established CNN baselines. Across these settings, DPNs achieve strong accuracy while reducing model complexity or computational and memory costs.

  • Experimental setup: DPNs are evaluated on ImageNet-1k, Places365-Standard, and PASCAL VOC across classification, detection, and semantic segmentation tasks.The experiments use standard benchmark datasets and assess whether DPNs generalize beyond image classification.
  • ImageNet-1k classification: 0.5% lower top-1 error than ResNeXt-101 (32 × 4d) and 1.5% lower than DenseNet-161 are achieved by DPN-92 with considerably fewer FLOPs.These comparisons concern ImageNet-1k single-crop validation error rates.
  • ImageNet-1k classification: 25% fewer FLOPs and a 236 MB model size are reported for DPN-98 while it surpasses ResNeXt-101 (64 × 4d) on ImageNet-1k.DPN-131 further exceeds Very Deep PolyNet in single-model accuracy with a 304 MB model versus 365 MB.
  • Training cost: 15% faster training and 9% less memory are reported for DPN-98 than the best-performing ResNeXt, while DPN-131 trains about 2 times faster than Very Deep PolyNet.The actual-cost comparison uses a single node with four K80 GPUs; theoretical DPN-98 computational cost is 25% lower than the best-performing ResNeXt.
  • Places365-Standard: DPN-92 achieves the best Places365 validation accuracy with 138 MB versus 163 MB for the compared model.The experiment uses DPN-92 rather than deeper DPN variants on the Places365-Standard scene classification dataset.
  • PASCAL VOC detection: 82.5% mAP is achieved on PASCAL VOC 2007 object detection, improving by 6.1% over ResNet-101 and 2.4% over ResNeXt-101 (32 × 4d).The comparison replaces only the residual backbone within a ResNet-based Faster R-CNN framework.
  • PASCAL VOC segmentation: DPN-92 obtains the highest overall mIoU on PASCAL VOC 2012 semantic segmentation, improving 1.7% over ResNet-101 and more than 3 times the 0.5% improvement attributed to ResNeXt-101 (32 × 4d).The models are evaluated within a DeepLab-ASPP-L framework.

6 Conclusion

The paper connects residual and densely connected networks through a HORNN interpretation and uses that connection to construct DPN. Across classification, detection, and segmentation, the resulting architecture combines high accuracy with efficiency and broader task applicability.

  • Conclusion: DPNs combine residual and densely connected path benefits, supporting feature re-usage, low resource consumption, and effective representation learning across several vision tasks.The conclusion reports benefits for image classification, object detection, and semantic segmentation, including application by replacing the base network.
  • Conclusion: The paper interprets densely connected networks through Higher Order RNNs and characterizes residual networks as densely connected networks with shared connections.This interpretation motivates the proposed dual path architecture.

A Testing with Mean-Max Pooling

Mean-Max Pooling is a lightweight testing-time modification that combines global average and max pooling before softmax. It improves testing accuracy across all evaluated models, including images larger than training crops.

  • 0.5 * (global average pooling + global max pooling) defines the Mean-Max Pooling layer inserted before the final softmax.The technique first converts a trained CNN into a convolutional network, then adds this layer during testing.
  • Testing accuracy improves for all evaluated models when Mean-Max Pooling is used.
Loading 1707.01629v2…