Source-linked AI summary
NAS-Bench-101: Towards Reproducible Neural Architecture Search
Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, Frank Hutter
TL;DR
NAS research is difficult to reproduce because existing methods can require enormous computation and differ in training procedures and search spaces. NAS-Bench-101 addresses this by providing a public tabular benchmark built from evaluated CNN architectures, enabling comprehensive search-space analysis and rapid benchmarking of NAS algorithms.
Problem
NAS methods can require months of computation and are difficult to compare because they use different training procedures and search spaces.
Method
NAS-Bench-101 trains and evaluates many CNN architectures on CIFAR-10, compiling their metrics into a public table for querying instead of costly retraining.
Results
The benchmark supports comprehensive analysis of an exhaustively evaluated search space and fast benchmarking of diverse NAS optimization algorithms.
Takeaways & Limitations
NAS-Bench-101 makes NAS research more accessible and reproducible while preserving the original architecture optimization problem for rigorous algorithm comparison.
Takeaways & Limitations
The evaluated models do not reach current state-of-the-art CIFAR-10 performance because the search space and training setup are constrained.
Abstract
from arXiv · showhide
Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public architecture dataset for NAS research. To build NAS-Bench-101, we carefully constructed a compact, yet expressive, search space, exploiting graph isomorphisms to identify 423k unique convolutional architectures. We trained and evaluated all of these architectures multiple times on CIFAR-10 and compiled the results into a large dataset of over 5 million trained models. This allows researchers to evaluate the quality of a diverse range of models in milliseconds by querying the pre-computed dataset. We demonstrate its utility by analyzing the dataset as a whole and by benchmarking a range of architecture optimization algorithms.
1. Introduction
NAS-Bench-101 addresses reproducibility and accessibility problems in neural architecture search by providing a public, exhaustively evaluated architecture dataset. It standardizes architectures and training evaluation so search methods can be analyzed and benchmarked through precomputed results.
- NAS research is difficult to reproduce because some methods require months of computation and differing training procedures and search spaces hinder fair comparison.
- NAS-Bench-101 maps 423k unique CNN architectures to metrics including runtime and accuracy, enabling experiments through table queries instead of retraining models.The dataset and training code are fully public.
- NAS-Bench-101 supports comprehensive search-space analysis and fast benchmarking of evolutionary, random, and Bayesian optimization methods.Search runs rank architectures by validation accuracy and are scored using mean test regret after a training-time budget.
- The benchmark exhaustively evaluates a compact cell-based search space designed to retain ResNet-like and Inception-like structures while remaining tractable.The space uses three operations, at most seven vertices, and at most nine edges.
- Each architecture is trained three times across four budgets, with training, validation, and testing accuracy, training time, and parameter count recorded.
3. NASBench as a Dataset
NAS-Bench-101 enables exhaustive analysis of architecture performance, cost, design choices, topology, and locality across its search space. The dataset reveals strong accuracy distributions, operation and topology trade-offs, and locality around high-performing architectures.
- Dataset statistics: Most architectures exceed 90% validation and test accuracy, while the best reaches 94.32% mean test accuracy.ResNet-like and Inception-like cells reach 93.12% and 92.95%, respectively.
- Dataset statistics: Training time, parameter count, and validation accuracy are positively correlated, but topology and operation selection also determine accuracy-cost trade-offs.ResNet- and Inception-like cells lie near the accuracy-over-cost Pareto frontier, while the best cell is not the most computationally intensive.
- Architectural design: Replacing 3 × 3 convolution with 1 × 1 convolution lowers validation accuracy by 1.16% while reducing relative training time by 14.11%.Replacing it with 3 × 3 max-pooling lowers accuracy by 1.99% and reduces training time by 9.84%.
- Architectural design: Mean validation accuracy appears optimal at depth 3 and increases with width through 5, while training time generally rises with depth and width.Width 1 networks are an exception because their feed-forward structure avoids channel splitting under the combine semantics.
- Locality: Accuracy correlations remain high at short edit distances but become indistinguishable from noise beyond about distance 6.Locality also appears around the global maximum, whose basin has a coarse-grained width of about 6.
- Locality: Only about 1 in 50,000 random samples reaches the top graphs, yet 35.4% of the search space lies within distance 6 of one.The top graphs include the best architecture and other architectures within twice its mean standard error, including graph isomorphisms.
4. NASBench as a Benchmark
NAS-Bench-101 provides a reproducible tabular benchmark for comparing NAS and HPO algorithms, making large-scale algorithm evaluation feasible without retraining models. Across 500 trials, regularized evolution, BOHB, and SMAC outperformed random search after roughly 50,000 TPU seconds, while generalization tests preserved algorithm rankings across search-space sizes.
- The benchmark compares random search, regularized evolution, SMAC, TPE, Hyperband, and BOHB using publicly available implementations.
- 500 independent trials per algorithm became feasible through tabular evaluation; otherwise, the experiments would have required over 900 TPU years.
- RE, BOHB, and SMAC began outperforming random search after roughly 50,000 TPU seconds and reached random search’s final performance about five times faster.
- TPE fell back to random-search performance, while Hyperband and BOHB lacked the speedups often observed over random search or Bayesian optimization.
- The benchmark’s generalization experiment compared algorithms on NAS-Bench-Mini, containing 64.5k cells, and NAS-Bench-101, containing 423k cells.
- Algorithm rankings remained consistent across the smaller and larger spaces, with RS < NRE < RE, and across regularized-evolution hyperparameter settings.
5. Discussion
The discussion examines how fixed hyperparameters approximate per-architecture optimization and reports strong agreement between the approximation and fully optimized performance. It also bounds the benchmark’s absolute accuracy because its compact search space and standardized training pipeline omit several modern architectural and training enhancements.
- Hyperparameter optimization selects numerical and categorical training settings for each architecture, while pure NAS seeks the best architecture under optimal hyperparameters.
- Because per-architecture inner HPO is computationally intractable, NAS-Bench-101 approximates it using hyperparameters estimated by maximizing average accuracy over a small architecture subset.
- A fixed hyperparameter configuration produced strongly correlated architecture rankings with fully optimized hyperparameters, achieving a Spearman rank correlation of 0.9155.
- The best architectures were ranked especially consistently under the fixed and fully optimized hyperparameter evaluations.
- NAS-Bench-101 models do not reach current CIFAR-10 state-of-the-art performance because the search space and training pipeline omit larger models, long augmentation-based training, and advanced regularization.
6. Conclusion
NAS-Bench-101 is an inexpensive tabular benchmark that preserves the NAS optimization problem, enabling rapid and rigorous algorithm comparisons while supporting large-scale search-space analysis and more accessible, reproducible research.
- NAS-Bench-101 preserves the original NAS optimization problem while making architectures inexpensive to evaluate.The benchmark supports rapid comparison without the enormous computational budgets often used in NAS projects.
- The dataset enables analysis of an exhaustively evaluated set of convolutional neural architectures at unprecedented scale.
- Open-sourcing the data and generation code aims to make NAS research more accessible and reproducible.
- The authors position NAS-Bench-101 as the first in a continually improving sequence of rigorous benchmarks for NAS.
S1. Identifying Isomorphic Cells
The search space contains computationally equivalent cell representations, so NAS-Bench-101 identifies isomorphic cells and evaluates only one canonical representative per equivalence class.
- Isomorphic cells have different adjacency matrices or labels but encode the same computation.Figure 1 illustrates two differently represented cells with identical computation.
- Vertices outside any input-to-output path can be pruned without changing a cell’s effective network behavior.
- Iterative graph hashing combines neighboring-vertex information and vertex labels to produce an isomorphism-invariant fixed-length hash.
- The benchmark enumerates unique cells, selects one canonical cell per equivalence class, and queries canonical results by hashing proposed cells.This avoids repeating expensive training and evaluation for computationally equivalent graph representations.
S2. Implementation Details
The implementation uses shared training settings and standardized algorithm protocols, with selected hyperparameters and public implementations supporting controlled benchmarking across NAS methods.
- Training hyperparameters were tuned on 50 randomly sampled cells and generally do not significantly affect cell rankings within reasonable ranges.
- All methods use the same encoding structure, while non-random methods receive well-performing hyperparameters selected by grid search.
- Random search samples architectures uniformly from all possible configurations in the configuration space.
- Regularized evolution mutates sampled edges or operators and uses population size PS=100 with tournament size TS=10 for final evaluation.
- TPE, Hyperband, BOHB, SMAC, and reinforcement learning use specified public implementations or evaluated settings, including η=3 for Hyperband and learning rate 0.5 for reinforcement learning.Changing Hyperband’s η changes budgets not included in NAS-Bench-101; SMAC uses five trees for final evaluation.
S3. Encoding
An alternative continuous encoding enforces the maximum-edge constraint by construction, but its continuous structure limits straightforward use of some NAS algorithms and changes their observed ranking.
- The alternative encoding assigns each possible edge a value p_i ∈ [0, 1] and selects the N highest-valued edges, with N ∈ 0, ..., 9.Operator encoding remains unchanged.
- The encoding guarantees that no architecture violates the maximum of nine edges.
- Its continuous search space makes regularized evolution and reinforcement learning difficult to apply without major changes.
- Under this encoding, TPE outperforms SMAC and BOHB, although the ranking changes relative to the default encoding.The authors attribute the possible discrepancy to SMAC and BOHB hyperparameters optimized for the default encoding.
S4. REINFORCE Baseline Approach
The paper benchmarks REINFORCE alongside evolutionary and random-search methods, finding it competitive with non-regularized evolution but weaker than regularized evolution. A separate PPO-based RL controller fails to learn effectively without additional tuning.
- RL comparison: The PPO-based LSTM controller tends to converge quickly to a far-from-optimal configuration without additional hyperparameter tuning.The authors suspect the space's conditional structure, including invalid cells with more than 9 edges, contributes to this behavior.
- Method: REINFORCE samples edge and operation choices from multinomial distributions to generate new architectures.The controller outputs distributions for each of 21 possible edges and 5 operations.
- Results: REINFORCE performs around as strongly as non-regularized evolution, while both are weaker than regularized evolution.All three methods outperform the random-search baseline.
S5. The NAS-HPO-Bench Datasets
NAS-HPO-Bench complements NAS-Bench-101 by evaluating hyperparameter configurations for feedforward networks across four non-image regression domains. Its smaller graph space includes hyperparameter choices, enabling analysis of their interactions and relative importance.
- Dataset scope: NAS-HPO-Bench contains 62208 hyperparameter configurations of 2-layer feedforward networks across four non-image regression domains.The configurations vary hidden units, activation types, dropout, learning rate, batch size, and learning-rate schedule.
- Dataset scope: Unlike NAS-Bench-101, NAS-HPO-Bench includes hyperparameter choices directly in its search space.This enables measurement of hyperparameter interactions and relative importance.
- Comparison: NAS-HPO-Bench has a much smaller graph space than NAS-Bench-101 but is complementary because it includes hyperparameter variation.
S6. Guidelines for Future Benchmarking of Experiments on NAS-Bench-101
The paper recommends reproducible, resource-aware, and statistically robust NAS-Bench-101 evaluations. These practices include repeated runs, resource-based performance plots, test-set isolation, seed-robustness analysis, and hyperparameter sensitivity checks.
- Experimental protocol: Run each NAS algorithm many times; the paper uses 500 runs in its experiments.
- Experimental protocol: Plot performance against estimated wall-clock time or function evaluations to compare algorithms under different resource constraints.Algorithms should track the best architecture found at each time step.
- Experimental protocol: Select architectures using only training and validation data during search, reserving test error for offline evaluation after runs finish.
- Robustness: Plot empirical cumulative distributions across many runs to assess robustness to random-number-generator seeds.
- Hyperparameters and encodings: Use the same NAS-Bench-101 hyperparameter settings as on other benchmarks, and report performance under alternative settings for sensitivity analysis.The guidelines also recommend studying alternative architecture encodings when applicable.