PaperlayerView exact revision on arXiv

Source-linked AI summary

Designing Network Design Spaces

Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, Piotr Dollár

arXiv:2003.13678v1cs.CVcs.LG

TL;DR

Network design traditionally struggles to produce simple, generalizable principles, while NAS typically returns a single setting-specific network. The paper designs network design spaces at the population level, deriving RegNet from network-structure exploration through a quantized linear parameterization. RegNet works across compute regimes and, under comparable training settings and flops, outperforms EfficientNet while being up to 5× faster on GPUs.

  • Problem

    NAS produces a single network tuned to a specific setting, leaving a need for simple network models and design principles that generalize across settings.

  • Method

    The paper progressively refines an unconstrained design space into simpler populations and applies this methodology to network structure, producing RegNet with quantized-linear stage widths and depths.

  • Results

    Under comparable training settings and flops, RegNet models outperform EfficientNet models while being up to 5× faster on GPUs.

  • Takeaways & Limitations

    Designing network design spaces is presented as a promising avenue for discovering interpretable principles and simple, fast networks that work across compute regimes.

  • Takeaways & Limitations

    The study focuses on network structure as the simplest design-space component, while richer spaces that include operators may yield better networks.

Abstract

from arXiv · show

In this work, we present a new network design paradigm. Our goal is to help advance the understanding of network design and discover design principles that generalize across settings. Instead of focusing on designing individual network instances, we design network design spaces that parametrize populations of networks. The overall process is analogous to classic manual design of networks, but elevated to the design space level. Using our methodology we explore the structure aspect of network design and arrive at a low-dimensional design space consisting of simple, regular networks that we call RegNet. The core insight of the RegNet parametrization is surprisingly simple: widths and depths of good networks can be explained by a quantized linear function. We analyze the RegNet design space and arrive at interesting findings that do not match the current practice of network design. The RegNet design space provides simple and fast networks that work well across a wide range of flop regimes. Under comparable training settings and flops, the RegNet models outperform the popular EfficientNet models while being up to 5x faster on GPUs.

1. Introduction

The paper introduces design space design, which designs parametrized populations of networks rather than individual instances, aiming for interpretable principles that generalize across settings. Applied to network structure, the process produces RegNet, a simple low-dimensional space whose widths and depths follow a quantized linear function.

  • Motivation: The paradigm addresses NAS's limitation of producing a single network tuned to a specific setting rather than generalizable design principles.The stated aim is to find models that are simple, understandable, buildable, and generalizable.
  • Contribution: The method designs parametrized network populations while combining manual design's interpretability with NAS's semi-automated procedures.The design-space terminology emphasizes designing the space itself rather than searching within it for one instance.
  • Design Space Design: Design space design refines an initial network design space into simpler or better-performing populations of models.The process is analogous to sequential manual network design but operates at the population level.
  • RegNet: RegNet is a low-dimensional design space of regular networks whose stage widths and depths are determined by a quantized linear function.Compared with AnyNet, RegNet has simpler, more interpretable models and a higher concentration of good models.
  • Findings: The best RegNet models have stable depth across compute regimes, at approximately 20 blocks, and use neither bottleneck nor inverted-bottleneck blocks.The design space is developed in a low-compute, low-epoch ImageNet setting and tested across larger compute regimes, schedules, and block types.
  • Results: Under comparable training settings and flops, RegNet models outperform EfficientNet models while being up to 5× faster on GPUs.The comparisons also report improvements over standard ResNet and ResNeXt models across all metrics.

2. Related Work

Related work spans manual network design, automated search for network instances, scaling rules for different compute regimes, and distribution-level comparison of network populations. This paper instead emphasizes designing novel design spaces, which can complement NAS and support structural insights.

  • Manual network design: Manual network design advanced accuracy by discovering architectural choices such as greater depth and residual connections.Examples include AlexNet, VGG, Inception, ResNet, ResNeXt, DenseNet, and MobileNet.
  • Automated network design: Most NAS work searches for the best network instances within a fixed, manually designed design space.The paper's focus differs by targeting the design of novel design spaces themselves.
  • Automated network design: Better design spaces can complement NAS by improving search efficiency and enlarging the set of potentially better models.This establishes design-space design as complementary to, rather than a replacement for, NAS search algorithms.
  • Network scaling: Network scaling commonly adapts a single instance across regimes by varying depth, width, resolution, or these dimensions jointly.The related-work passage notes that single-instance procedures leave adaptation to different flop regimes unclear.
  • Comparing networks: The paper adopts distribution-level comparison of sampled network populations to guide design-space analysis and discover general design principles.This methodology aligns with the goal of reasoning about populations rather than isolated network instances.
  • Parameterization: The quantized linear parameterization is distinguished by empirical justification and insights into structural choices such as stage block counts.The paper contrasts these contributions with prior work sharing similarities in setting stage widths.

3. Design Space Design

The paper designs network design spaces rather than individual models, progressively simplifying an unconstrained population while preserving or improving its quality. This process yields RegNet, whose quantized linear structure concentrates strong models, improves random-search efficiency, and generalizes across settings.

  • Motivation: Design space design studies populations of models to discover principles that generalize beyond a single model tuned for one setting.The approach treats a design space as a model population analyzed through its error distribution rather than searching only for the best instance.
  • Method: The methodology progressively simplifies an unconstrained design space while maintaining or improving its quality.Each design step refines the input space into a simpler or better-performing population, guided by distribution estimates.
  • AnyNet refinement: Sharing bottleneck ratios and group widths simplifies AnyNet without materially changing its error distribution, while increasing stage widths and depths improves the EDF.The shared-parameter refinements reduce degrees of freedom and expose analyzable trends; monotonic width and depth constraints further improve results.
  • Linear structure: Top models are well described by a quantized linear width parameterization, and lower fitting error efit is associated with better network error.The best models have fitting error near 0, while average efit improves from AnyNetXC to AnyNetXE.
  • RegNet: RegNet reduces AnyNetX from 16 to 6 dimensions and nearly 10 orders of magnitude in design-space size while improving average error and retaining the best models.RegNet generates widths and depths from a six-parameter structure and maintains diversity for different settings.
  • Generalization: RegNetX remains better than AnyNetXE and AnyNetXA at higher flops, longer schedules, five stages, and varied block types.The consistent ordering RegNetX > AnyNetXE > AnyNetXA indicates generalization to the tested settings.

4. Analyzing the RegNetX Design Space

RegNetX analysis identifies simple structural trends that motivate constrained, efficient design spaces and useful alternate choices. These findings include stable depth, no bottleneck, bounded complexity, and gains from SE.

  • RegNet trends: RegNetX analysis reveals design trends that differ from common deep-network practice and support simple models.The analysis examines fine-grained network behavior using 100 sampled models trained for 25 epochs.
  • RegNet trends: Best-model depth remains stable across flop regimes at approximately 20 blocks, or 60 layers.This contrasts with the common practice of increasing depth for higher flop regimes.
  • RegNet trends: The best models use bottleneck ratio b = 1.0, while their width multiplier is approximately 2.5.The remaining parameters g, wa, and w0 increase with complexity.
  • Complexity analysis: RegNetX complexity analysis treats activations as an important measure because they can strongly affect accelerator runtime.The analysis considers activations alongside flops and parameters, especially for memory-bound hardware accelerators.
  • RegNetX constrained: Constraining RegNetX with b = 1, d ≤40, wm ≥2, and limits on parameters and activations yields faster, lower-parameter, lower-memory models without affecting accuracy.The constrained variant is reported as superior across all flop regimes.
  • Alternate design choices: Inverted bottlenecks and depthwise convolution degrade the EDF relative to b = 1 and g ≥1, whereas RegNetY with SE yields gains.Varying resolution also harms results in the reported comparison.

5. Comparison to Existing Networks

Controlled comparisons show that RegNet models are effective across mobile, standard-baseline, and broad compute regimes. Under comparable training settings, they improve on ResNe(X)t and generally outperform EfficientNet while offering faster execution.

  • Mobile regime: RegNet models achieve surprisingly good results in the approximately 600MF mobile regime despite using a basic training setup.The comparison uses originally reported errors for existing models and RegNet models trained in the basic setup.
  • Mobile regime: RegNet models use a 100-epoch schedule with weight decay but no other regularization, providing simple baselines for future work.Most mobile networks use longer schedules and enhancements such as deep supervision, Cutout, DropPath, or AutoAugment.
  • ResNe(X)t comparisons: Under identical components and training settings, RegNetX improves on standard ResNet and ResNeXt models across all complexity metrics.The comparison isolates gains attributed to the RegNetX design space and network structure.
  • ResNe(X)t comparisons: RegNetX models are effective under fixed activation, inference-time, training-time, and flop budgets.Activations are emphasized because they can strongly influence runtime on accelerators such as GPUs.
  • Comparison methodology: EfficientNet comparisons reproduce the exact EfficientNet models under RegNet’s 100-epoch schedule with weight decay only to isolate architecture effects.Learning rate and weight decay are optimized, and the setup is shared with RegNet for fair comparison.
  • EfficientNet comparisons: Under comparable training settings, RegNetY outperforms EfficientNet for most flop regimes, while RegNetX and RegNetY perform better at higher flops.EfficientNet performs better than RegNetY at low flops, whereas RegNetY is better at intermediate flops.
  • EfficientNet comparisons: RegNetX-8000 is 5× faster than EfficientNet-B5 while having lower error.EfficientNet activations scale linearly with flops, compared with square-root scaling for RegNet models, contributing to slower GPU training and inference.

6. Conclusion

The paper introduces network design spaces as a promising paradigm, with RegNet providing simple models that perform well across compute regimes.

  • The authors present designing network design spaces as a promising avenue for future research.
  • RegNet models achieve favorable comparisons with existing networks on ImageNetV2, including RESNE(X)T and EFFICIENTNET.

Appendix A: Test Set Evaluation

On ImageNetV2, model rankings are generally preserved, although performance gaps narrow and absolute errors increase. RegNet remains competitive across flop regimes and up to 5× faster than EfficientNet on GPUs.

  • ImageNetV2 preserves overall model rankings, but absolute errors increase on the new test set.
  • RESNE(X)T comparisons: RegNetX models remain favorable against RESNE(X)T across flop regimes, including low-compute settings where strong RESNE(X)T models are unavailable.
  • EfficientNet comparisons: RegNet models perform comparably to state-of-the-art EfficientNet while being up to 5× faster on GPUs.
  • Additional comparisons: At 400MF, RegNetY performs better with Swish than ReLU, whereas at 6.4GF ReLU performs better.

Appendix B: Additional Ablations

Additional ablations show that fixed-depth networks can match variable-depth networks across flop regimes, while fewer stages and inverted bottlenecks degrade performance at high compute. Activation-function effects depend on compute and convolution type.

  • Fixed depth: Fixed-depth networks with d = 20 match variable-depth networks across all flop regimes in both average and best-case performance.These networks also match the best results reported in the main text.
  • Fewer stages: Three-stage networks perform considerably worse at 6.4GF than the compared RegNet configurations.The authors note that changes to the stem or head may be needed for three-stage networks.
  • Inverted bottleneck: At 6.4GF, using an inverted bottleneck with b < 1 degrades results further.This extends the earlier observation that inverted bottlenecks degrade performance in a low-compute regime.
  • Swish vs. ReLU: Swish outperforms ReLU at low flops, while ReLU is better at high flops; with depthwise convolution, Swish performs much better.The underlying reason for the favorable interaction between depthwise convolution and Swish is unclear.
  • Optimization settings: Optimization sweeps vary learning rate and weight decay across RegNetX populations and complexity regimes to identify stable training settings.The sweep uses bootstrap analysis and includes lr·wd as a combined quantity.

Appendix C: Optimization Settings

The authors tune RegNet optimization settings and repeat the procedure for EfficientNet to support fair comparisons. Training enhancements substantially affect EfficientNet-B0 results, underscoring the importance of controlling training setup.

  • RegNet settings: RegNet experiments use lr = 0.1 and wd = 5·10^-5, with linear learning-rate scaling and five-epoch gradual warmup when increasing batch size.
  • EfficientNet settings: Learning rate and weight decay remain stable across EfficientNet complexity regimes under the repeated optimization sweep.These settings were used to support fair comparisons with RegNet.
  • Training enhancements: Without training enhancements, EfficientNet-B0 results are approximately 2% lower, highlighting the importance of carefully controlling training setup.Enhancements include DropPath and longer training, while the original setup additionally used RM-SProp and AutoAugment.
  • Block types: The X block performs best among the four block types tested in the generalization experiments.For the V block, residual connections provide no gain.

Appendix D: Implementation Details

The appendix describes implementation choices for adapting RegNet across block types and compute regimes. It covers group-width sampling and compatibility, alternative blocks, robustness to block-specific parameter differences, and Y-block details.

  • Group width compatibility: Widths are adjusted when incompatible with group widths by setting g = w when g > w or rounding w to a divisible value.The resulting width differs from the original by at most 1/3; bottleneck widths are adjusted similarly.
  • Group width ranges: Group widths are increased for higher-compute regimes; at 3.2GF, the range is 16 ≤ g ≤ 64 with g divisible by 8.The lower-compute example samples g ≤ 32.
  • Block types: The RegNet design space generalizes to R, V, and VR blocks in addition to the X block.R removes groups from X, V uses one 3×3 convolution, and VR adds residual connections to V.
  • Block types: Good parameter values can differ across block types, such as b > 1 being better than b = 1 for R blocks, while the approach remains robust.This contrast is stated relative to the X block.
  • Y block details: The Y block adds an SE operation after the X block’s 3×3 convolution and uses an SE reduction ratio of 1/4.The authors report comparable performance for the choices they tested.
Loading 2003.13678v1…