PaperlayerView exact revision on arXiv

Source-linked AI summary

NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size

Xuanyi Dong, Lu Liu, Katarzyna Musial, Bogdan Gabrys

arXiv:2009.00437v6cs.LGstat.ML

TL;DR

NAS results are difficult to compare because studies use different search spaces and training setups. NATS-Bench unifies topology and size search with standardized architecture evaluations and diagnostics, revealing dataset-dependent rankings and architecture-performance variation.

  • Problem

    Different NAS studies use different search spaces, augmentation, regularization, schedulers, and hyperparameters, making algorithm performance less comparable.

  • Method

    NATS-Bench defines topology and size search spaces using fixed cell-based architectures with densely connected DAGs, and provides training diagnostics for each architecture.

  • Results

    Across datasets, architecture performance varies with parameters, FLOPs, and operation connectivity, while rankings can decrease in cross-dataset transfer and differ from broader-search-space observations.

  • Takeaways & Limitations

    NATS-Bench supports more comparable and computationally cost-friendly NAS evaluation while enabling analysis of architecture choices, transferability, and diagnostic information.

  • Takeaways & Limitations

    Whether empirical observations from the benchmark generalize to other scenarios remains under investigation, and cross-dataset transfer does not guarantee good performance.

Abstract

from arXiv · show

Neural architecture search (NAS) has attracted a lot of attention and has been illustrated to bring tangible benefits in a large number of applications in the past few years. Architecture topology and architecture size have been regarded as two of the most important aspects for the performance of deep learning models and the community has spawned lots of searching algorithms for both aspects of the neural architectures. However, the performance gain from these searching algorithms is achieved under different search spaces and training setups. This makes the overall performance of the algorithms to some extent incomparable and the improvement from a sub-module of the searching model unclear. In this paper, we propose NATS-Bench, a unified benchmark on searching for both topology and size, for (almost) any up-to-date NAS algorithm. NATS-Bench includes the search space of 15,625 neural cell candidates for architecture topology and 32,768 for architecture size on three datasets. We analyze the validity of our benchmark in terms of various criteria and performance comparison of all candidates in the search space. We also show the versatility of NATS-Bench by benchmarking 13 recent state-of-the-art NAS algorithms on it. All logs and diagnostic information trained using the same setup for each candidate are provided. This facilitates a much larger community of researchers to focus on developing better NAS algorithms in a more comparable and computationally cost friendly environment. All codes are publicly available at: https://xuanyidong.com/assets/projects/NATS-Bench.

1 INTRODUCTION

NATS-Bench addresses incomparable NAS evaluations by unifying topology and size search under consistent architectures and training information. It defines complementary search spaces and supplies reusable evaluations for fairer algorithm comparison.

  • Different search spaces, training strategies, and validation splits make the relative contributions of NAS algorithms difficult to compare.
  • Existing benchmarks mainly study topology, while architecture size—which significantly affects model performance—is largely omitted.
  • NATS-Bench introduces topology and size search spaces designed to broaden benchmark coverage and improve NAS reproducibility.
  • Topology candidates vary operations on edges in densely connected cells, while size candidates vary channel counts across layers.
  • The benchmark contains 15,625 topology candidates and 32,768 size candidates, each trained repeatedly on three datasets.
  • Shared training logs and diagnostic information reduce repeated training and validation costs, helping researchers focus on search algorithms.

2 RELATED WORK

Related benchmarks improve NAS evaluation, but prior work has emphasized topology and leaves broader reproducibility, generalization, and size analysis insufficiently covered. NATS-Bench extends this benchmarking scope with larger, deeper, multi-dataset evaluation.

  • Reported NAS improvements remain difficult to attribute because search spaces, optimization setups, and evaluation procedures differ across studies.
  • NAS-Bench-101 is a large-scale architecture dataset, but its unmodified benchmark applies only to selected NAS algorithms.
  • NATS-Bench covers both topology and size, whereas NAS-Bench-101 provides only a topology search space.
  • Prior benchmarks have rarely addressed network size and optimizer effects together with NAS reproducibility and generalization.
  • NATS-Bench evaluates train, validation, and test performance across three datasets, supporting assessment of NAS algorithm generality.
  • NAS-HPO-Bench evaluates 62,208 joint configurations but contains only 144 architectures, limiting its coverage for comparing NAS algorithms.

3 NATS-BENCH

NATS-Bench is an algorithm-agnostic benchmark built from cell-based architectures, standardized training across three datasets, and detailed performance and diagnostic records. It supports both topology and size search while enabling efficient querying and analysis.

  • NATS-Bench: NATS-Bench is designed to apply to almost any up-to-date NAS algorithm and evaluates architectures on three datasets.
  • Architectures in the Search Space: Each architecture uses a fixed macro skeleton containing three cell stacks and residual down-sampling blocks, with identical cell topology throughout.
  • Architectures in the Search Space: Topology candidates are densely connected four-node DAG cells whose edges independently select among five representative operations.
  • Architecture Performance: The benchmark standardizes dataset splits and training strategies, including Nesterov momentum SGD, cosine learning-rate decay, and multiple training durations.
  • Architecture Performance: Performance records include loss and accuracy after every epoch, plus parameter counts and FLOPs, with results queryable through an API.
  • Diagnostic Information: Diagnostic information includes computational costs and fine-grained learning curves that can support constrained search, accuracy prediction, and analysis of convergence or overfitting.
  • What/Who can Benefit from NATS-Bench?: Querying stored results replaces repeated candidate training with low-cost database access, reducing approximation and validation burdens for NAS experiments.
  • What/Who can Benefit from NATS-Bench?: Benchmarking topology or size separately can clarify the effectiveness of algorithms targeting either search dimension and inform their intersection.

4 ANALYSIS OF NATS-BENCH

Across NATS-Bench, architecture performance depends on both computational scale and design choices, while rankings transfer differently across topology and size spaces. Validation rankings track test rankings closely, but cross-dataset transfer and trial variability limit straightforward selection.

  • Architecture Performance: Performance changes with parameter count and FLOPs, yet architectures with identical costs can still achieve different accuracy.This indicates that operation choices and their connections matter beyond model cost alone.
  • Architecture Performance: ResNet remains competitive, but the best candidates exceed it by about 2% on CIFAR-100 and ImageNet-16-120.At comparable parameter counts, the gap is about 1% on those datasets.
  • Architecture Performance: Pyramid-shaped size candidates are far from parameter–accuracy Pareto optimality but close to FLOPs–accuracy Pareto optimality.The evaluated channel configurations are 8-16-24-32-40, 8-16-32-48-64, and 32-40-48-56-64.
  • Architecture Ranking: Topology candidates show generally consistent rankings across the three datasets, whereas size-candidate rankings differ substantially.The differing size rankings indicate that optimal architecture sizes are dataset-dependent.
  • Architecture Ranking: Validation-based rankings resemble test-based rankings in both search spaces, allowing brute force to approximate the highest test accuracy.This establishes an empirical upper bound for NAS algorithms evaluated in the benchmark.
  • Architecture Ranking: Kendall rank correlation decreases sharply when only top candidates are retained, and directly transferring the best architecture between datasets cannot guarantee good performance.The authors identify this as motivation for transferable NAS algorithms beyond the vanilla strategy.

5 BENCHMARK

NATS-Bench evaluates NAS methods across standardized topology and size search spaces, datasets, and training procedures. Its experiments compare multi-trial and weight-sharing strategies while documenting efficiency, robustness, and limitations of the benchmark setup.

  • Evaluation setup: 13 recent NAS algorithms are evaluated on the topology and size search spaces using the NATS-Bench API.The benchmark includes random search, evolutionary, reinforcement-learning, differentiable, and hyperparameter-optimization methods.
  • Evaluation setup: Each experiment runs 500 times on three datasets under time budgets adjusted for dataset-specific training costs.CIFAR-10 uses a maximum budget of 2e4 seconds, with larger budgets for the other datasets.
  • Topology search results: On topology search, most methods perform similarly on CIFAR-10, while BOHB converges fastest on ImageNet-16-120 and REA leads BOHB on CIFAR-100 before 2e4 seconds.The reported CIFAR-100 ordering at 4e4 seconds is REA ≥ BOHB ≥ REINFORCE ≥ RANDOM.
  • Size search results: On size search, REA significantly outperforms the other methods across all datasets and increasingly approaches the best architecture as search time grows.The authors relate this behavior to REA fitting the prior that larger models tend to have higher accuracy without model-cost constraints.
  • Size search results: Channel-wise interpolation finds better models faster than masking-based strategies because it implicitly compares two channel candidates per layer at each search step.Masking strategies evaluate only one candidate during each search step.
  • Caveats: The benchmark’s empirical observations may not generalize because implementations and hyperparameters can differ from original methods and may be sub-optimal for NATS-Bench.The authors explicitly state that some algorithms remain sensitive to hyperparameters and that generalization to other scenarios is under investigation.

6 DISCUSSION

The discussion addresses fair, reproducible NAS evaluation, benchmark over-fitting, diagnostic uses, generalization, and limitations or extensions of NATS-Bench.

  • Benchmark usage: NATS-Bench recommends fixed meta-hyperparameters, provided performance, and multiple search runs to reduce dataset-specific over-fitting and improve comparison stability.The benchmark also warns against structural restrictions tailored to known best architectures.
  • Limitations: Using the same training hyperparameters for every architecture can bias performance because optimal configurations may vary across architectures.Jointly optimizing architectures and hyperparameters remains computationally expensive and open.
  • Future directions: NATS-Bench can benchmark cost-constrained NAS because latency and FLOPs information are available.This supports evaluating search algorithms with different cost losses.
  • Future directions: NATS-Bench supplies diagnostic information that may support new parameter-sharing approaches balancing search efficiency against architecture-specific accuracy.The discussion raises whether learned parameters from many models can combine benefits of multi-trial and weight-sharing methods.
  • Generalization: Benchmark observations show potential generalization: REA ≥ REINFORCE ≥ RANDOM and GDAS ≥ DARTS (2nd) ≥ DARTS (1st), consistent with prior benchmarks.The paper states that generalization is not guaranteed.

APPLICATION PROGRAMMING INTERFACE (API)

The API provides constructors, iteration, search, configuration, cost, and performance queries for architectures in the topology and size search spaces.

  • Creating and browsing benchmarks: Users create a benchmark instance by selecting either the topology or size search space.The API exposes both benchmark variants through the `create` function.
  • Creating and browsing benchmarks: The benchmark object can be iterated to print each architecture’s index and structure.The example tracks the total number of architectures with `amount = len(nats_bench)`.
  • Searching and querying: The API finds the highest validation accuracy for a chosen dataset and training setting.The example searches CIFAR-100 validation accuracy after specifying 90-epoch training.
  • Searching and querying: Users can query an architecture’s configuration, cost information, and performance across datasets.Available information includes latency, training time, parameter count, and validation accuracy.
  • Detailed results: Detailed results can be queried by architecture index, dataset, and epoch setting, with values organized by random seed.The example retrieves CIFAR-100 results for architecture 284 after 12 epochs and evaluates seed 888.

COMPARISON UNDER DIFFERENT TRAINING EPOCHS

Training-fidelity comparisons show that shorter training can be competitive in topology search, while longer training eventually helps in size search.

  • Experimental setup: The experiment compares REA using 12 epochs with longer 200-epoch or 90-epoch training to examine efficiency–accuracy trade-offs.REA is selected as the best-performing multi-trial algorithm in the benchmark.
  • Topology search: For topology search on CIFAR-10, REA with H0 (12 epochs) converges after 1e5 seconds, while H1 (200 epochs) remains worse after 1.2e6 seconds.The comparison concerns REA’s discovered architectures under different training budgets.
  • Size search: For size search, REA with H0 (12 epochs) converges after 2e4 seconds, while H2 (90 epochs) overtakes H0 after about 8e4 seconds.A similar pattern occurs on CIFAR-100 and ImageNet-16-120.
Loading 2009.00437v6…