PaperlayerView exact revision on arXiv

Source-linked AI summary

NAS evaluation is frustratingly hard

Antoine Yang, Pedro M. Esperança, Fabio M. Carlucci

arXiv:1912.12522v3cs.LGcs.CVstat.ML

TL;DR

NAS lacks a shared protocol for comparing methods and attributing gains to search strategies rather than engineered spaces or training procedures. This paper benchmarks eight methods across five datasets with a relative-improvement baseline and studies DARTS components. It finds generally small gains over random architectures, with training protocol and macro-structure having greater impact than searched operations.

  • Problem

    NAS methods lack a shared evaluation protocol and sufficient ablations, leaving their comparative effectiveness and component contributions unclear.

  • Method

    The paper benchmarks 8 NAS methods on 5 datasets using relative improvement over randomly sampled architectures, then performs DARTS ablations of training and structural components.

  • Results

    Most NAS methods rarely substantially outperform the average randomly sampled architecture, while training protocol has a greater impact on accuracy than the architecture itself.

  • Takeaways & Limitations

    NAS comparisons should use random-architecture baselines, multiple datasets, ablations, and reproducible evaluation practices.

  • Takeaways & Limitations

    Architecture rankings can depend strongly on the random seed and the number of cells, complicating comparisons when search and final models differ in depth.

Abstract

from arXiv · show

Neural Architecture Search (NAS) is an exciting new field which promises to be as much as a game-changer as Convolutional Neural Networks were in 2012. Despite many great works leading to substantial improvements on a variety of tasks, comparison between different methods is still very much an open issue. While most algorithms are tested on the same datasets, there is no shared experimental protocol followed by all. As such, and due to the under-use of ablation studies, there is a lack of clarity regarding why certain methods are more effective than others. Our first contribution is a benchmark of $8$ NAS methods on $5$ datasets. To overcome the hurdle of comparing methods with different search spaces, we propose using a method's relative improvement over the randomly sampled average architecture, which effectively removes advantages arising from expertly engineered search spaces or training protocols. Surprisingly, we find that many NAS techniques struggle to significantly beat the average architecture baseline. We perform further experiments with the commonly used DARTS search space in order to understand the contribution of each component in the NAS pipeline. These experiments highlight that: (i) the use of tricks in the evaluation protocol has a predominant impact on the reported performance of architectures; (ii) the cell-based search space has a very narrow accuracy range, such that the seed has a considerable impact on architecture rankings; (iii) the hand-designed macro-structure (cells) is more important than the searched micro-structure (operations); and (iv) the depth-gap is a real phenomenon, evidenced by the change in rankings between $8$ and $20$ cell architectures. To conclude, we suggest best practices, that we hope will prove useful for the community and help mitigate current NAS pitfalls. The code used is available at https://github.com/antoyang/NAS-Benchmark.

1 INTRODUCTION

The paper frames NAS as a way to automate architecture design, but inconsistent protocols and limited ablations make methods difficult to compare and their effective components unclear. It benchmarks eight methods and uses DARTS experiments to examine how search, training, architecture structure, and evaluation choices affect reported performance.

  • NAS aims to replace manually designed architectures, whose task-specific tuning is tedious and expensive.
  • NAS methods combine a search space, search strategy, and model evaluation strategy, each of which can affect final performance.
  • Inconsistent protocols and limited ablation studies make it difficult to identify which components produce reported gains.
  • The study benchmarks 8 NAS algorithms on 5 computer-vision datasets using relative improvement over the average architecture as a comparison metric.
  • DARTS experiments test whether training protocol, architecture choice, hyperparameters, random seed, and macro-structure drive accuracy and rankings.

2 RELATED WORK

Prior work identifies inadequate baselines, excessive method complexity, incomplete ablations, and missing reproducibility details as recurring NAS problems. This paper extends those concerns by comparing against random sampling and dissecting the DARTS search space and training techniques.

  • Prior critiques identify weak baselines, excessive complexity without ablations, and insufficient reproducibility details in NAS studies.
  • The paper uses relative improvement over the average randomly sampled architecture to compare methods across search spaces.
  • Its DARTS study separates architecture effects from training techniques including Cutout, DropPath, and AutoAugment.
  • Related work reports that random search with early stopping and multiple seeds can be a competitive baseline, while engineered search spaces may make random architectures perform well.

3 NAS BENCHMARK

The benchmark evaluates NAS methods against random architectures across multiple datasets, using relative improvement to isolate search quality from shared search spaces and training protocols. Results show generally small gains over random sampling, narrow accuracy ranges, dataset-dependent variation, and substantial computational cost.

  • 3 NAS BENCHMARK: The benchmark evaluates 8 methods on 5 datasets to reveal general search-strategy trends rather than only the most accurate algorithm.
  • 3 NAS BENCHMARK: The protocol samples 8 random architectures and compares them with 8 searched architectures using each method’s augmentation code.
  • 3 NAS BENCHMARK: RI = 100 × (Accm − Accr)/Accr measures the search method’s accuracy relative to random sampling within the same search space and training protocol.
  • 3.2 RESULTS: Across 5 datasets, improvements over random sampling are generally small, and some methods perform below the average randomly sampled architecture.
  • 3.2 RESULTS: On CIFAR10, PDARTS, MANAS, DARTS, and StacNAS perform similarly and positively relative to their search spaces, whereas other datasets show more variance.
  • 3.2 RESULTS: Search computational cost is a limiting factor in NAS applicability and is evaluated alongside search-phase performance on CIFAR10.

4 COMPARISON OF TRAINING PROTOCOLS

Training protocol choices can change DARTS accuracy far more than architecture selection, so reported NAS gains may primarily reflect evaluation tricks rather than superior search.

  • Methodology: The study compares randomly sampled DARTS architectures under training protocols ranging from a 600-epoch Base setup to a trick-enhanced full protocol.The evaluation samples eight architectures, trains them under different protocols, and reports mean, standard deviation, and maximum top-1 accuracy.
  • Results: Over 3 percentage points separate the simplest and most advanced protocols, exceeding the best 0.69 p.p. improvement over random sampling.The best eight random architectures under the strongest protocol reached 98.15%, only 0.25 p.p. below the cited state of the art.
  • Results: The training protocol is often more important than the architecture used for final accuracy.The authors report similar findings when applying the same protocols to ResNet-50.
  • Results: Training-curve and accuracy-distribution analyses further examine how randomly sampled architectures behave during training.Figure 4 presents curves for 214 architectures and an inset histogram of accuracies at different epochs.

5 STUDY OF DARTS’ SEARCH SPACE

The DARTS search space produces a narrow accuracy range, making seeds and cell counts influential in architecture rankings, while macro-structure matters more than individual operations.

  • Architecture variability: Moving-average accuracies correlate more strongly with final accuracy than raw accuracies, supporting their use for early stopping.The correlation with final accuracy grows slowly and approximately linearly across epochs.
  • Operations: The classical and intentionally sub-optimal operation spaces differ by only 0.18 accuracy points, indicating that specific operations are not the key ingredient.The comparison uses 214 models from the DARTS space and 56 models from the modified space under the same training protocol.
  • Ranking stability: Changing the seed heavily affects rankings: the Kendall tau correlation is 0.48, with mean accuracy changes of 0.13% ± 0.08 and a maximum change of 0.39%.These shifts are substantial relative to the small gap between random architectures and NAS methods.
  • Ranking stability: Changing network depth also alters rankings, with Kendall tau 0.54 and shifts of up to 18 positions among 32 architectures.The authors note that weight-sharing methods may experience an even stronger effect when search and final models use different cell counts.
  • Implications: Because seed effects are large, the authors recommend averaging final accuracy over multiple seeds and relate ranking instability to lucky initialization.They also connect depth mismatch and the lottery ticket hypothesis to difficulty improving over random architectures.

6 DISCUSSION AND BEST PRACTICES

The paper recommends evaluation and reporting practices that separate search quality from training and search-space design, while improving reproducibility and cross-dataset validity.

  • Augmentation tricks: Results should be reported both with and without training tricks because engineered protocols can hide the search algorithm’s contribution.The recommendation follows experiments showing that training protocols can dominate architecture effects.
  • Search space: NAS evaluations should measure randomly sampled architectures, for example by reporting the mean and standard deviation of k samples.Random sampling establishes the average model quality and is distinct from random search as a search strategy.
  • Multiple datasets: Future work should test NAS methods across datasets with different image sizes, sample counts, class granularity, and learning tasks.The authors identify single-dataset focus as risking algorithmic overfitting or heavy dependence on hyperparameter tuning.
  • Hidden components: Ablation studies should isolate contributions from operations, macro-structure, and training protocol within the NAS pipeline.The DARTS experiments indicate that macro-structure and training protocol contribute more than the selected operations.
  • Reproducibility: For reproducibility, authors should release the best architecture, its seed when applicable, code, and detailed training hyperparameters.The paper also points to NAS-Bench-101 as a potentially useful architecture–accuracy resource.
  • Hyperparameter tuning cost: Hyperparameters should either generalize across tasks without tuning or have their tuning cost included in the NAS search budget.The paper identifies hyperparameter tuning as an extremely costly component of NAS.

7 CONCLUSIONS

The paper finds that NAS results are strongly shaped by training protocols and unstable search spaces, and recommends more reproducible, fair evaluation practices.

  • The authors recommend releasing architectures with seeds, code, and detailed training protocols to improve reproducibility.
  • Training protocols can affect final accuracy more than the selected network architecture.
  • Many NAS search spaces make architectures perform similarly while their relative rankings can shift easily.
  • A more general automated architecture-design approach is needed to reduce reliance on human expert knowledge in search spaces.

A APPENDIX

The appendix documents dataset-specific hyperparameters and reports that search spaces were kept unchanged while network sizes were tuned for several datasets.

  • Search spaces were left unchanged across the experiments.
  • Hyperparameters were selected close to the original papers and sometimes updated to newer implementations.
  • Network size was tuned similarly across methods for SPORT8, MIT67, and FLOWERS102.
  • All experiments were run on NVIDIA Tesla V100 GPUs.

A.1 METHODS AND HYPERPARAMETERS

The methods appendix specifies shared evaluation procedures and implementation details for the eight NAS algorithms, including training, search, and architecture-evaluation settings.

  • During search, models use training and validation subsets separately; final evaluation trains on training plus validation data and tests on the test subset.
  • All eight methods share optimizer-related settings including momentum values and gradient clipping at 5.
  • DARTS-derived methods share common hyperparameters and implementation patterns.
  • DARTS uses cutout, path dropout with probability 0.2, and an auxiliary tower weighted by 0.4.
  • PDARTS progressively reduces operations from 8 to 5 to 3 while increasing skip-connect dropout across search stages.
  • NSGANET searches a micro search space with 2 cells, 9 operations, and 5 blocks per cell on 8-layer networks.
  • NAO encodes and decodes architectures with LSTM models before evaluating the resulting CNN architecture at a larger depth.
  • ENAS trains shared network parameters and controller policy parameters during search, then evaluates the searched architecture in an expanded cell network.

A.2 DATASETS

The study evaluates image-classification datasets spanning object, action, scene, and fine-grained recognition, using dataset-specific splits and preprocessing.

  • CIFAR10 contains 10 classes with 50,000 training images and 10,000 test images of size 32×32.
  • CIFAR100 contains 100 classes with 50,000 training images and 10,000 test images of size 32×32.
  • CIFAR10 and CIFAR100 are split into training, validation, and testing subsets of 25,000, 25,000, and 10,000 images.
  • SPORT8 contains 1,579 images across 8 sport-event categories, stressing NAS generalization because of its small size.
  • MIT67 has 67 indoor-scene classes and 15,620 images, while FLOWERS102 has 102 flower-species classes and 8,189 images.
  • SPORT8, MIT67, and FLOWERS102 use 40/40/20% train, validation, and test splits with image augmentation and resizing procedures.

A.3 ADDITIONAL RESULTS

Figure 10 extends Figure 3 with CIFAR10 results from training ResNet-50 alongside DARTS under the same comparison framework.

  • Figure 10 reports CIFAR10 results for ResNet-50 and DARTS across 8 runs of each training protocol.Darker bars represent ResNet-50, while lighter bars represent DARTS.
  • For ResNet-50, the auxiliary tower was added after layer 2.
  • Because DropPath was not straightforward to apply, the ResNet-50 experiments used Stochastic Depth instead for a similar effect.
Loading 1912.12522v3…