PaperlayerView exact revision on arXiv

Source-linked AI summary

PolyNet: A Pursuit of Structural Diversity in Very Deep Networks

Xingcheng Zhang, Zhizhong Li, Chen Change Loy, Dahua Lin

arXiv:1611.05725v2cs.CV

TL;DR

The paper asks whether depth or width is the best route to optimal performance and explores structural diversity as an alternative design dimension. It introduces PolyInception modules and designs Very Deep PolyNet, achieving lower ILSVRC 2012 top-5 error than Inception-ResNet-v2.

  • Problem

    The paper questions whether increasing network depth or width is the best strategy for achieving optimal performance and identifies structural diversity as an underexplored alternative.

  • Method

    The paper develops PolyInception, polynomial combinations of Inception units arranged in parallel or cascaded paths and usable as modular network building blocks.

  • Results

    Very Deep PolyNet achieves 4.25% single-crop and 3.45% multi-crop top-5 error on ILSVRC 2012, versus Inception-ResNet-v2's 4.9% and 3.7%.

  • Takeaways & Limitations

    The study reports that enhancing structural diversity alongside depth can further improve performance over Inception-ResNet-v2 and deeper variants.

  • Takeaways & Limitations

    The architecture is designed under a specified training constraint: fitting a small cluster of 32 GPUs with 12GB RAM each and batch size 512.

Abstract

from arXiv · show

A number of studies have shown that increasing the depth or width of convolutional networks is a rewarding approach to improve the performance of image recognition. In our study, however, we observed difficulties along both directions. On one hand, the pursuit for very deep networks is met with a diminishing return and increased training difficulty; on the other hand, widening a network would result in a quadratic growth in both computational cost and memory demand. These difficulties motivate us to explore structural diversity in designing deep networks, a new dimension beyond just depth and width. Specifically, we present a new family of modules, namely the PolyInception, which can be flexibly inserted in isolation or in a composition as replacements of different parts of a network. Choosing PolyInception modules with the guidance of architectural efficiency can improve the expressive power while preserving comparable computational cost. The Very Deep PolyNet, designed following this direction, demonstrates substantial improvements over the state-of-the-art on the ILSVRC 2012 benchmark. Compared to Inception-ResNet-v2, it reduces the top-5 validation error on single crops from 4.9% to 4.25%, and that on multi-crops from 3.7% to 3.45%.

1. Introduction

The paper argues that depth and width have important limitations, motivating structural diversity as another network-design dimension. It introduces PolyInception modules and the Very Deep PolyNet, which improves ILSVRC validation error over Inception-ResNet-v2.

  • Depth improvements eventually diminish, while widening causes quadratic growth in runtime complexity and memory demand.The study reports that widening’s computational increase may not be justified by its performance gain.
  • Structural diversity is explored as an alternative network-design dimension beyond depth and width.The paper identifies unanswered questions about how to build diversity into networks and when it may be more rewarding than pursuing depth.
  • PolyInception combines Inception units polynomially through parallel or cascaded paths to enhance structural diversity.The modules can be inserted alone or composed to replace existing network modules.
  • Very Deep PolyNet uses carefully selected PolyInception configurations across three spatial-resolution stages.The architecture is designed through systematic comparisons of different design choices.
  • 4.25% single-crop and 3.45% multi-crop top-5 error improve on Inception-ResNet-v2’s 4.9% and 3.7%, respectively.The model also uses 92M parameters versus 133M for a deepened Inception-ResNet-v2 under the same computational budget.
  • The paper presents structural diversity, PolyInception, and Very Deep PolyNet as its major contributions.Very Deep PolyNet is reported to obtain the best single-network performance on ILSVRC.

2. Related Work

Related work shows that progress has relied on increasing depth, width, residual connections, ensembles, and richer network topologies. PolyNet builds on these structural-diversity ideas while using Inception-ResNet-v2 as its base model.

  • ImageNet networks progressed from AlexNet to ResNet with increasingly deep architectures, but very deep networks can be difficult to train and prone to overfitting.Residual structures mitigate these issues through identity paths that pass signals forward and backward.
  • Wide residual networks and related variants improve residual representation through increased width, nested residuals, or multiple parallel blocks.These studies challenge depth as the sole source of residual-network performance.
  • Residual structures can be viewed as exponential ensembles of relatively shallow networks.Stochastic depth and Swapout explicitly use training procedures that form ensembles through random layer or unit selection.
  • DenseNet and FractalNet explore non-linear topologies by adding dense shortcuts or using truncated fractal layouts.These approaches extend exploration beyond sequential network formation.
  • Inception-ResNet-v2 combines Inception and residual structures and serves as PolyNet’s base model.Inception blocks capture residuals and derive from the fundamental components of GoogleNet.

3. PolyInception and PolyNet

The paper explores structural diversity as an alternative to simply increasing depth or width, introducing PolyInception modules and evaluating their use in PolyNet. Carefully selected and mixed PolyInception designs improve performance with different computational and parameter costs.

  • Motivation: Beyond 100 residual units, increasing ResNet depth yields diminishing performance returns.Increasing from 16 to 89 units improves performance substantially, whereas 166 units doubles cost for only a 0.1% error reduction, from 4.9% to 4.8%.
  • Motivation: Structural diversity is proposed as a network-design dimension beyond depth and width.The motivation follows observed limitations of deeper and wider networks and unresolved questions about how to build diversity into networks.
  • PolyInception Modules: PolyInception modules combine Inception units through polynomial compositions using parallel or cascaded paths.They generalize additive residual combinations and can increase expressive power by sending inputs through deeper transformations before merging paths.
  • PolyInception Modules: The cascaded form of poly-2 is mathematically equivalent to its parallel form while using only 2/3 of the computational cost.Poly-2 shares parameters between its two Inception blocks, whereas mpoly-2 does not and therefore doubles parameter size while providing stronger expressive power.
  • An Ablation Study: Replacing Inception residual units with PolyInceptions improves performance but increases computational cost, with stage-specific efficiency trade-offs.Stage B benefits most from mpoly-3, while poly-3 uses only 1/3 of its parameter size; stages A and C show smaller gains and favor k-way designs.

4. Training PolyNets

PolyNet training combines data augmentation, initialization by insertion, residual scaling, and adaptive stochastic paths to stabilize optimization and control overfitting in very deep networks.

  • Data augmentation: Training uses multi-area augmentation with random crops covering 8%–100% of each image, aspect ratios from 3/4 to 4/3, resizing to 299 × 299, and 50% horizontal flipping.Very Deep PolyNet receives 331 × 331 inputs.
  • Training settings: Experiments use RMSProp, batch normalization, scheduled learning-rate reductions, and 32-bit computation across multi-GPU servers.IR 3-6-3 models use single nodes with mini-batches of 256, while IR 6-12-6 models use double-node clusters with the same batch size.
  • Initialization: Initialization by insertion transfers parameters from a smaller pretrained network while randomly initializing newly inserted units.The strategy is intended to stabilize and accelerate convergence in very deep networks.
  • Residual scaling: Residual scaling replaces I + F + G with I + βF + βG using β = 0.3 to reduce training instability.The residual paths are dampened before being added to module outputs.
  • Stochastic paths: Stochastic paths randomly drop module paths, producing smaller distinct PolyInception subnetworks during training.The method is activated adaptively after serious overfitting is observed, with dropping probabilities increasing from 0 to 0.25 from bottom to top.

5. Overall Comparison

Very Deep PolyNet is compared with deeper Inception-ResNet and other state-of-the-art architectures using single- and multi-crop ImageNet evaluations. It achieves lower reported validation errors than Inception-ResNet-v2 while showing complementary benefits from depth and structural diversity.

  • Evaluation: Top-30% pooling aggregates scores from 36 crops at each of 8 scales for the multi-crop evaluation.Scores are averaged across the selected crops within each scale and then across scales.
  • Evaluation caveats: Reported results primarily use the authors’ implementations, which omit auxiliary loss and label smoothing and create a 0.2% top-5 accuracy gap from the published Inception-ResNet-v2 result.Table 1 includes some exceptions, and non-blacklisted validation subsets may yield slightly optimistic error rates.
  • Depth versus diversity: The experiments compare progressively deepened Inception-ResNets with PolyNets ranging from PolyNet 3-6-3 to the final mixed-configuration Very Deep PolyNet.Adaptive stochastic depth is applied to ameliorate overfitting in very deep models.
  • Overall comparison: 4.25% and 3.45% top-5 validation errors are achieved by Very Deep PolyNet under single-crop and multi-crop evaluation, respectively.These results are compared with Inception-ResNet-v2 at 4.9% and 3.7%.
  • Overall comparison: Very Deep PolyNet outperforms ResNet, Inception-ResNet-v2, and their very deep variants on ImageNet validation and test evaluations.The comparison includes a Very Deep Inception-ResNet constructed by stacking more residual units.
  • Depth versus diversity: The multi-crop performance of ResNet-500 is slightly lower than ResNet-269, indicating that greater depth alone does not necessarily improve performance.The paper uses this comparison to distinguish deepening from enhancing structural diversity.

6. Conclusion

The paper explores structural diversity through PolyInception modules and uses systematic experiments to design Very Deep PolyNet. The resulting model outperforms Inception-ResNet-v2 and deeper variants, supporting joint emphasis on depth and diversity.

  • Conclusion: Structural diversity is proposed as a new direction for designing deep networks beyond depth alone.The study systematically examines PolyInception and uses the findings to design Very Deep PolyNet.
  • Conclusion: Very Deep PolyNet outperforms Inception-ResNet-v2 and its deeper variants in the reported experiments.The conclusion attributes further performance improvement to enhancing structural diversity along with increasing depth.
Loading 1611.05725v2…