PaperlayerView exact revision on arXiv

Source-linked AI summary

NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search

Xuanyi Dong, Yi Yang

arXiv:2001.00326v2cs.CV

TL;DR

NAS comparisons are difficult because methods use different search spaces and training setups. NAS-Bench-201 provides a fixed, cell-based benchmark with reusable results across three datasets and diagnostic information, while its analyses include consistent rankings across related benchmarks.

  • Problem

    Different NAS search spaces and training configurations make performance comparisons difficult.

  • Method

    NAS-Bench-201 fixes a four-node, five-operation cell search space and provides evaluations, logs, and diagnostics for its architecture candidates across three datasets.

  • Results

    The benchmark evaluates 15,625 architectures across three datasets and reports consistent rankings with NAS-Bench-101 and NAS-Bench-1SHOT1 for several NAS method groups.

  • Takeaways & Limitations

    NAS-Bench-201 enables fairer and more computationally efficient NAS comparisons while supporting analysis of search behavior and future algorithm designs.

  • Takeaways & Limitations

    NAS-Bench-201 uses the same training hyper-parameter configuration for all architectures, which may bias some architecture performances.

Abstract

from arXiv · show

Neural architecture search (NAS) has achieved breakthrough success in a great number of applications in the past few years. It could be time to take a step back and analyze the good and bad aspects in the field of NAS. A variety of algorithms search architectures under different search space. These searched architectures are trained using different setups, e.g., hyper-parameters, data augmentation, regularization. This raises a comparability problem when comparing the performance of various NAS algorithms. NAS-Bench-101 has shown success to alleviate this problem. In this work, we propose an extension to NAS-Bench-101: NAS-Bench-201 with a different search space, results on multiple datasets, and more diagnostic information. NAS-Bench-201 has a fixed search space and provides a unified benchmark for almost any up-to-date NAS algorithms. The design of our search space is inspired from the one used in the most popular cell-based searching algorithms, where a cell is represented as a DAG. Each edge here is associated with an operation selected from a predefined operation set. For it to be applicable for all NAS algorithms, the search space defined in NAS-Bench-201 includes all possible architectures generated by 4 nodes and 5 associated operation options, which results in 15,625 candidates in total. The training log and the performance for each architecture candidate are provided for three datasets. This allows researchers to avoid unnecessary repetitive training for selected candidate and focus solely on the search algorithm itself. The training time saved for every candidate also largely improves the efficiency of many methods. We provide additional diagnostic information such as fine-grained loss and accuracy, which can give inspirations to new designs of NAS algorithms. In further support, we have analyzed it from many aspects and benchmarked 10 recent NAS algorithms.

1 INTRODUCTION

NAS research faces comparability problems because algorithms use different search spaces and training setups. NAS-Bench-201 addresses this with a fixed cell-based benchmark, broad applicability, reusable training results, multiple datasets, and diagnostic information.

  • Motivation: Different search spaces and training setups make NAS algorithm comparisons difficult.Variations include macro skeletons, cell operation sets, augmentation, regularization, schedulers, and hyper-parameters.
  • Benchmark design: NAS-Bench-201 extends prior benchmarks with a fixed cell search space designed for broad NAS applicability.It transforms architecture search into searching a cell within a predefined skeleton.
  • Benchmark benefits: The benchmark provides a unified environment for comparing recent NAS algorithms while reducing architecture-training and validation overhead.Researchers can focus on search algorithms rather than repetitive training or hyper-parameter tuning.
  • Benchmark benefits: NAS-Bench-201 supplies full training logs, results on multiple datasets, and diagnostic information for analyzing NAS behavior.The authors also systematically analyze the search space and benchmark 10 recent NAS algorithms.

2 NAS-Bench-201

NAS-Bench-201 defines a cell-based architecture space and evaluates every candidate under unified settings across multiple datasets. It exposes performance, computational-cost, fine-grained training, and parameter information for NAS research.

  • Benchmark scope: NAS-Bench-201 is algorithm-agnostic and evaluates architectures on CIFAR-10, CIFAR-100, and ImageNet-16-120.The benchmark uses consistent training and evaluation splits for NAS algorithms.
  • Macro skeleton: Each architecture uses a fixed macro skeleton containing three stacks of five searched cells separated by residual downsampling blocks.The skeleton begins with a 3-by-3 convolution and ends with global average pooling and softmax classification.
  • Searched cell: Each searched cell is a densely connected four-node DAG whose edges select among five operations: zeroize, skip connection, 1-by-1 convolution, 3-by-3 convolution, and 3-by-3 average pooling.Nodes sum the feature maps transformed by incoming edges.
  • Training and evaluation: Every architecture is trained with a shared hyper-parameter strategy based on Nesterov momentum SGD for 200 epochs.The setup uses cross-entropy loss, weight decay of 0.0005, and cosine learning-rate annealing; CIFAR-10 also has a 12-epoch configuration.
  • Training and evaluation: The benchmark provides training, validation, and test loss and accuracy, queryable for each trial with negligible computational cost.These records let researchers reuse evaluations and focus on NAS search algorithms.
  • Diagnostic information: NAS-Bench-201 additionally provides parameter count, FLOPs, latency, fine-grained epoch-level trajectories, and trained parameters.These resources support computationally constrained search, convergence analysis, hypernetwork supervision, and architecture mutation without retraining from scratch.

3 DIFFERENCE WITH EXISTING NAS BENCHMARKS

NAS-Bench-201 extends existing NAS benchmarks with an algorithm-agnostic cell search space and additional diagnostic information, addressing limitations in applicability and scale.

  • NAS-Bench-201 associates operations with edges, whereas NAS-Bench-101 defines operation candidates on nodes.
  • NAS-Bench-101 is only directly applicable to selected algorithms because its edge constraint is difficult to incorporate into parameter-sharing methods.
  • NAS-Bench-201 sacrifices the number of nodes and includes all possible edges, making its search space algorithm-agnostic.
  • NAS-Bench-201 provides extra information including architecture computational cost and fine-grained training and evaluation time.
  • NAS-HPO-Bench evaluated 144 candidate architectures, which may be insufficient for evaluating different NAS algorithms.

4 ANALYSIS OF NAS-Bench-201

NAS-Bench-201 reveals how architecture performance, rankings, and validation–test correlations behave across datasets and during training. Its analyses expose both cross-dataset regularities and limits on direct transfer.

  • Different parameter counts affect architecture performance, while architectures with the same parameter count can also perform differently.The analysis attributes the latter observation to how operations and cells are connected.
  • ResNet remains competitive but is about 2% behind the best architecture on CIFAR-100 and ImageNet-16-120.It is also about 1% behind the best architecture with the same parameter count on those datasets.
  • Architecture rankings are generally consistent across CIFAR-10, CIFAR-100, and ImageNet-16-120, with slightly different variance.
  • Within-dataset validation–test correlations are higher than cross-dataset correlations.The correlations are reported across six validation and test sets covering the three datasets.
  • Directly transferring the best architecture from one dataset to another cannot 100% secure good performance.This finding motivates better transferable NAS algorithms instead of a vanilla transfer strategy.
  • Validation-based rankings gradually converge to rankings based on final test accuracy as training proceeds.

5 BENCHMARK

NAS-Bench-201 benchmarks ten NAS methods and reports their search behavior, efficiency, and sensitivity to batch-normalization strategies. Fine-grained benchmark information enables rapid evaluation, while some methods remain sensitive to implementation and hyper-parameter choices.

  • Ten recent NAS algorithms are evaluated, covering random search, evolutionary, reinforcement-learning, differentiable, and parameter-sharing methods.
  • NAS-Bench-201 reduces search for methods without parameter sharing to seconds, although parameter-sharing methods still require several GPU hours.
  • Methods without parameter sharing outperform others, possibly because short-budget training with a converged scheduler preserves architecture rankings.
  • Batch statistics outperform running estimates for batch-normalization layers in parameter-sharing NAS methods.
  • DARTS progressively overfits to all-skip architectures, while DARTS-V2 alleviates this behavior somewhat and batch statistics further reduce it.
  • RSPS and SETN exhibit higher searched-architecture variance than GDAS.
  • Reported algorithm results may be non-optimal because hyper-parameters can be poorly matched to NAS-Bench-201.

6 DISCUSSION

The discussion addresses benchmark overfitting, diagnostic-information opportunities, cross-search-space generalization, and the unresolved challenge of jointly optimizing architectures and training hyper-parameters.

  • Benchmark-specific regularization can overfit NAS-Bench-201 and may fail to generalize to other benchmarks.
  • Users are encouraged to use the provided performance and report multiple search runs for fairer, more stable comparisons.
  • Using one hyper-parameter configuration for every architecture can bias performance because optimal training settings may vary by architecture.
  • Jointly searching architectures and optimal hyper-parameter configurations remains computationally expensive and open.
  • NAS-Bench-201’s diagnostic information may motivate improved parameter-sharing designs using information from 15,625 trained models.
  • Observed rankings agree with NAS-Bench-101 and NAS-Bench-1SHOT1, providing preliminary evidence that some findings generalize across search spaces.

7 CONCLUSION & FUTURE WORK

The paper introduces NAS-Bench-201 as a reproducible NAS benchmark with broad algorithm coverage, multi-dataset evaluations, diagnostic information, and baselines for future work.

  • NAS-Bench-201 evaluates 15,625 architectures on three datasets and reports multiple performance metrics.
  • The benchmark supports direct evaluation of almost any NAS algorithm and includes analyses and baselines for future research.
  • Future work will consider joint hyper-parameter optimization and a much larger search space.

A MORE DETAILS OF NAS-Bench-201

NAS-Bench-201 encodes every candidate as a six-operation vector, evaluates all encoded architectures without isomorphism reduction, and offers short-budget training results for bandit methods.

  • Each architecture is encoded by a 6-dimensional vector whose entries specify operations on the six cell edges.
  • The encoding yields 5^6 = 15625 total unique models, while isomorphism-based topology counts are smaller.
  • The benchmark trains and evaluates every architecture without considering isomorphism.
  • For CIFAR-10 bandit methods, short-budget performance is available from schedules converging at the 12th or 200th epoch.

B IMPLEMENTATION DETAILS

The benchmark re-implements 10 NAS algorithms under specified time budgets and algorithm-specific training configurations. It also evaluates parameter-sharing behavior and examines sensitivity to sample size and learning rate.

  • Algorithm implementations: 10 NAS algorithms are re-implemented from publicly available code to search architectures on NAS-Bench-201.The implementation details for each algorithm are provided.
  • Time budgets: 12000 seconds is used as the total time budget for RS, REINFORCE, ENAS, and BOHB, matching the first-order DARTS search-time baseline.The reported baseline is about 12000 seconds on CIFAR-10.
  • Time budgets: RSPS, GDAS, SETN, and ENAS run five times longer than DARTS because they optimize 1/5 of the parameters per iteration.Most hyper-parameters otherwise follow DARTS, and batch-normalization scale and shift parameters are not learned during cell search.
  • Sensitivity analyses: Figures 9 and 10 evaluate REA sample-size effects and REINFORCE learning-rate effects using CIFAR-10 validation accuracy of the searched architecture.Figure 9 concerns the CIFAR-10 validation set; Figure 10 reports searched-architecture validation accuracy.
  • Algorithm configurations: DARTS uses 50 shared-parameter training epochs, whereas RSPS and GDAS use 250 epochs and SETN selects 100 architectures after shared-parameter training.GDAS linearly decays its Gumbel-Softmax temperature from 10 to 0.1; RSPS evaluates 100 randomly selected architectures.
  • Algorithm configurations: BOHB finishes when simulated training time reaches 12000 seconds, while REA uses validation accuracy after 12 training epochs as fitness and sample sizes from [3, 5, 10].The sample-size choice is reported according to Figure 9.

C DISCUSSION FOR NAS WITH PARAMETER SHARING

Parameter sharing improves search efficiency by sharing weights across millions of candidate architectures, but the shared parameters cannot be optimal for all candidates. The benchmark evaluates this issue quantitatively using DARTS, GDAS, and SETN.

  • Parameter sharing: Shared parameters span millions of architecture candidates, making it almost impossible for them to be optimal for every candidate.The paper evaluates the trained shared parameters quantitatively through DARTS, GDAS, and SETN.

D DETAILED INFORMATION OF NAS-Bench-201

NAS-Bench-201 provides repeated training results, hardware-specific latency measurements, and a convenient API for querying architectures, metrics, costs, and trials. These interfaces expose both performance and computational information for benchmark use.

  • Benchmark data: 6219 architectures are trained once, 1621 twice, and 7785 three times with different random seeds in NAS-Bench-201 version 1.0.The authors state that all architectures are being actively trained with more seeds and that the benchmark will continue updating.
  • API usage: The API can be installed with “pip install nas-bench-201” and provides examples for enumerating architectures and querying architecture metadata.The examples show iteration over the API and retrieval of metrics for a selected architecture.
  • API usage: The API returns training loss and accuracy for datasets such as CIFAR-10 and exposes FLOPs, parameter count, and latency through computational-cost queries.The examples query CIFAR-10 training metrics and CIFAR-100 computational costs.
  • API usage: Users can query the index of a specific architecture, retrieve each trial’s results, and inspect how many trials exist for that architecture on CIFAR-100.The benchmark data file and additional usage examples are available online.
Loading 2001.00326v2…