PaperlayerView exact revision on arXiv

Source-linked AI summary

On Network Design Spaces for Visual Recognition

Ilija Radosavovic, Justin Johnson, Saining Xie, Wan-Yen Lo, Piotr Dollár

arXiv:1905.13214v1cs.CVcs.LG

TL;DR

Architecture research needs fairer ways to compare entire model families, not only selected models or error-complexity curves. The paper introduces statistically analyzed distributions of sampled design spaces and finds substantial differences among NAS spaces, while ResNeXt can be comparable to more complex NAS spaces.

  • Problem

    Existing architecture comparisons rely on point or curve estimates, leaving it difficult to compare unconstrained model families while controlling for confounding factors such as network complexity.

  • Method

    The paper samples models from parameterized design spaces and applies statistical techniques to their error and complexity distributions, complementing NAS by analyzing the design space itself.

  • Results

    NAS design spaces differ significantly, and standard model families such as ResNeXt can be comparable to the more complex design spaces used in recent NAS methods.

  • Takeaways & Limitations

    Design spaces should be treated as a core part of architecture development and evaluation, alongside search algorithms and individual model discovery.

Abstract

from arXiv · show

Over the past several years progress in designing better neural network architectures for visual recognition has been substantial. To help sustain this rate of progress, in this work we propose to reexamine the methodology for comparing network architectures. In particular, we introduce a new comparison paradigm of distribution estimates, in which network design spaces are compared by applying statistical techniques to populations of sampled models, while controlling for confounding factors like network complexity. Compared to current methodologies of comparing point and curve estimates of model families, distribution estimates paint a more complete picture of the entire design landscape. As a case study, we examine design spaces used in neural architecture search (NAS). We find significant statistical differences between recent NAS design space variants that have been largely overlooked. Furthermore, our analysis reveals that the design spaces for standard model families like ResNeXt can be comparable to the more complex ones used in recent NAS work. We hope these insights into distribution analysis will enable more robust progress toward discovering better networks for visual recognition.

1. Introduction

The paper argues that architecture comparisons need a statistically grounded methodology that evaluates sampled populations of models rather than a few points or curves. It introduces distribution estimates and applies them to NAS and standard model families.

  • 1. Introduction: Point estimates often ignore model complexity, while curve estimates trace error against complexity for only a handful of selected models.Curve comparisons can still vary confounding factors between model families or leave one family suboptimal.
  • 1. Introduction: The methodology addresses whether model families can be compared while varying all relevant hyperparameters, despite the resulting space containing vastly many possible models.Sampling makes empirical comparison feasible without exhaustive evaluation.
  • 1. Introduction: Distribution estimates compare sampled models from parameterized design spaces using statistical techniques while controlling for network complexity.This is intended to provide a more complete and unbiased view of the design landscape than point or curve estimates.
  • 1. Introduction: Training over 100,000 models on CIFAR and validation on ImageNet show that robust estimates are feasible with roughly 100 to 1000 sampled models per family.The study spans VGG, ResNet, and ResNeXt families.
  • 1. Introduction: NAS design spaces differ significantly, while standard families such as ResNeXt can be comparable to the more complex spaces used in recent NAS methods.The analysis characterizes design spaces themselves, complementing NAS methods that search for the best model within a family.
  • 1. Introduction: The paper releases code, baselines, and statistics for tested models to support comparisons with the studied design spaces.The authors position the methodology as a practical tool for continued architecture research.

2. Related Work

Related work frames this paper as part of a broader shift toward empirical, reproducible evaluation of neural architectures. It distinguishes design-space analysis from hyperparameter tuning, NAS search algorithms, and conventional complexity measures.

  • 2. Related Work: The paper complements reproducibility efforts by proposing a more robust methodology for evaluating visual-recognition architectures.This motivation parallels guidelines in related machine-learning research that aim to support continued progress.
  • 2. Related Work: Large-scale empirical studies are presented as important when rigorous theoretical understanding of deep networks is unavailable.Related language-modeling studies show that thoroughly exploring design spaces can change conclusions about which architectures perform best.
  • 2. Related Work: Rather than first tuning hyperparameters to compare model families, the paper proposes directly analyzing sampled design spaces with statistical methods.This addresses the practical difficulty of making fair comparisons across families.
  • 2. Related Work: Unlike most NAS work, which emphasizes search algorithms, this paper focuses on characterizing the network design space itself.NAS combines a design space with a search algorithm, whereas this work studies the former.
  • 2. Related Work: Network complexity is controlled using practical measures such as parameter counts and multiply-add operations rather than theoretical complexity notions that may not predict neural-network behavior.These measures are used to analyze design spaces while reducing confounding factors.
  • 2. Related Work: The referenced ResNet statistics provide an example of architecture and error reporting used for comparison.The table reports original and reproduced errors for reference models.

3. Design Spaces

The paper defines a design space as a concrete, parameterized subset of a broader model family and studies it by sampling and training models. Its experiments instantiate standard architectures with independently varied structural hyperparameters.

  • 3. Design Spaces: A design space specifies a model-family parameterization and allowable values for every hyperparameter, fully determining each network instantiation.This distinguishes a concrete design space from an abstract model family such as ResNet.
  • 3. Design Spaces: Because a design space may contain exponentially many models, the paper samples a fixed set, evaluates them, and analyzes the resulting model distribution statistically.The evaluated models effectively form datasets for empirical studies of design spaces.
  • 3. Design Spaces: For each of three stages, the design spaces independently select block counts d_i and channels per block w_i; ResNeXt also selects bottleneck ratios r_i and group counts g_i.The resulting model counts are (dw)^3 without groups and (dwrg)^3 with groups.
  • 3. Design Spaces: Standard model families include vanilla feedforward networks, ResNet, and ResNeXt, differing in residual connections and grouped convolutions.The corresponding design spaces use shared stem-stage-head structures with family-specific block types.
  • 3. Design Spaces: Models are generated by uniformly sampling allowable hyperparameter values, with experiments training 25,000 models per design space on CIFAR-10.The main study totals 100,000 trained models and restricts models by FLOPs or parameter limits to reduce computational load.

4. Proposed Methodology

The methodology compares sampled design-space distributions rather than isolated best models, using EDFs and KS tests while controlling for complexity. This exposes when apparent differences reflect unequal sampling or complexity distributions rather than architecture design.

  • 4.1. Comparing Distributions: 90% of trials favored the 1000-model set’s minimum error even though both sets came from the same design space, demonstrating that point estimates can mislead.The comparison uses 100 versus 1000 sampled models.
  • 4.1. Comparing Distributions: Empirical distribution functions summarize the fraction of sampled models below each error threshold, enabling direct comparison of entire error distributions.The KS statistic measures the maximum vertical discrepancy between two EDFs.
  • 4.1. Comparing Distributions: The KS test gave D = 0.079 and p-value = 0.60 for the imbalanced samples, supporting failure to reject that they came from the same distribution.This contrasts with the misleading minimum-error comparison.
  • 4.2. Controlling for Complexity: Complexity-conditioned comparisons weight model sets so their complexity distributions align, isolating error differences not explained by complexity alone.The approach can condition on parameters or flops.
  • 4.2. Controlling for Complexity: Controlling for parameters or flops removed most of the ResNeXt-A versus ResNeXt-B gap, indicating that much of the unnormalized difference reflected mismatched complexity distributions.A small normalized gap remained, potentially associated with ResNeXt-B’s wider models and more groups.
  • 4.4. Minimal Sample Size: 100 samples provided a reasonable lower bound for distribution comparisons, while 1000 samples enabled more precise comparisons with diminishing returns beyond 1000.At 100 samples D was about 0.1, and at 1000 it began to saturate; the bounds may vary in other circumstances.
  • 4.5. Feasibility: Training 500 CIFAR models required about 250 GPU hours, comparable to the roughly 192 GPU hours for a typical ImageNet ResNet-50 baseline.The paper frames distribution evaluation as computationally comparable to a point estimate on a larger task.

5. Case Study: NAS

The NAS case study compares five design spaces under controlled complexity and finds substantial differences in their error distributions and random-search efficiency. It also shows that ResNeXt design spaces can be competitive with NAS spaces under comparable normalization and training settings.

  • 5.1. Design Spaces: The study reproduces five NAS cell structures, standardizes network stacking, varies width and depth to align complexity ranges, and samples models uniformly.The five spaces correspond to NASNet, AmoebaNet, PNAS, ENAS, and DARTS; approximately 1,000 models are trained per space.
  • 5.2. Design Space Comparisons: DARTS is best overall among the five NAS design spaces, while NASNet and Amoeba are noticeably worse; ENAS and PNAS have crossing error distributions.The comparison uses normalized error EDFs, with the differences reaching max KS test D = 0.51 between DARTS and NASNet.
  • 5.2. Design Space Comparisons: Newer NAS design spaces outperform the earliest NASNet and Amoeba spaces, suggesting that improvements in design spaces may contribute to reported gains.The paper gives PNAS’s removal of five NASNet operations as an example of pruning the design space.
  • 5.2. Design Space Comparisons: With random search fixed, design-space differences alone produce clear differences in search efficiency, reinforcing the need to decouple search algorithms from design spaces.The ordering by random-search efficiency is consistent with the ordering of the normalized EDFs.
  • 5.3. Comparisons to Standard Design Spaces: ResNeXt-B is comparable to DARTS when normalized by parameters, while DARTS slightly outperforms it when normalized by FLOPs; ResNeXt-A is worse in both cases.The parameter-normalized comparison reports KS test D = 0.09.
  • 5.3. Comparisons to Standard Design Spaces: Under enhanced training settings, ResNeXt⋆ achieves similar error to DARTS⋆ at comparable complexity, while the original ResNet-110 is also competitive.The enhanced setup includes a 600-epoch schedule and additional regularization; the table compares selected higher-complexity models.

6. Conclusion

The paper presents design-space analysis as a methodology for comparing convolutional-network families and encourages treating design spaces as a core part of model development and evaluation. Supporting experiments report consistent training-hyperparameter regions, model behavior, and trends across reruns.

  • 6. Conclusion: The methodology is intended to support analysis beyond convolutional image classification, including other model types, domains, and tasks.The paper specifically mentions RNNs, NLP, and detection as possible applications.
  • 6. Conclusion: Model errors remain consistent across 100 reruns of selected models, with a clear gap between top- and mid-ranked models.The corresponding figure reports consistency across two selected models for each design space.

Appendix A: Supporting Experiments

Supporting experiments examine training choices, rerun variability, stratified complexity analysis, and ImageNet transfer. They find consistent trends across design spaces and report ImageNet results that closely follow CIFAR results.

  • Appendix A: Supporting Experiments: A half-period cosine schedule with lr = 0.1 and wd = 5e-4 is used after a 5k-model study finds consistent hyperparameter behavior across Vanilla, ResNet, and DARTS.The study trains sampled models for 100 epochs and varies learning rate and weight decay log-uniformly.
  • Appendix A: Supporting Experiments: Trend estimates are robust to rerun count, with KS test D < 0.009 for both Vanilla and ResNet.The qualitative trends remain consistent across design spaces and numbers of reruns.
  • Appendix A: Supporting Experiments: Stratifying models into complexity buckets produces trends similar to normalized EDF analysis and serves as a first step toward that estimate.Bucket analysis computes distributions independently within complexity bins rather than reweighting models into one normalized EDF.
  • Appendix A: Supporting Experiments: Training-error variability arises mainly from random seeds and floating-point nondeterminism, although model-ranking gaps can exceed this variance.The paper notes that fixing the random seed does not substantially reduce the overall variance.
  • Appendix A: Supporting Experiments: ImageNet results closely follow the CIFAR results, providing a supporting validation on another dataset.The appendix reports the ImageNet experiments as a replication of the study’s conclusions.

Appendix B: ImageNet Experiments

ImageNet experiments test whether the distribution-estimate methodology transfers from CIFAR and remains informative with approximately 100 models per design space. The results largely reproduce the CIFAR findings, including transferable NAS ordering, search-efficiency differences, and normalization-dependent comparisons with standard spaces.

  • Design spaces: The ImageNet experiments use the CIFAR model families with an ImageNet stem and adjusted NAS widths and depths, sampling approximately 100 models per design space.The study includes five NAS and two ResNeXt design spaces and deliberately uses the minimal sample count to stress-test the methodology.
  • NAS distribution comparisons: NAS design-space ordering is consistent between CIFAR and ImageNet, supporting transferability of CIFAR-developed spaces to ImageNet.The normalized EDF shapes also match their CIFAR counterparts.
  • NAS random search efficiency: Random-search efficiency ordering follows EDF ordering on ImageNet, and design-space differences alone produce performance differences.These findings are consistent with the CIFAR experiments and are reported in Figure 16b.
  • Comparison to standard design spaces: ResNeXt-B is similar to DARTS when normalized by parameters, whereas NAS design spaces outperform standard ones considerably when normalized by FLOPs.This comparison is shown in Figure 16c.
  • Discussion: ImageNet results closely follow CIFAR results, indicating that 100 models per design space are sufficient for applying the methodology in practice.The authors present ImageNet as a test case because these experiments were run once after the methodology was finalized.
Loading 1905.13214v1…