PaperlayerView exact revision on arXiv

Source-linked AI summary

NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search

Arber Zela, Julien Siems, Frank Hutter

arXiv:2001.10422v2cs.LGcs.CVcs.NEstat.ML

TL;DR

NAS weight-sharing methods make architecture search efficient, but their behavior is difficult to study reproducibly and proxy-model rankings may not reflect final architecture quality. The paper introduces reusable one-shot NAS and benchmarking frameworks built on NAS-Bench-101, then evaluates method behavior, hyperparameter sensitivity, and black-box comparisons. The resulting benchmarks support fair, statistically sound analysis and are intended to improve reproducibility and understanding of one-shot NAS mechanisms.

  • Problem

    NAS results are difficult to reproduce, NAS-Bench-101 cannot directly benchmark one-shot optimizers, and proxy-model rankings may not predict performance in larger evaluation models.

  • Method

    The paper maps one-shot search spaces to NAS-Bench-101 and provides a general framework for running, tracking, and fairly comparing one-shot NAS variants.

  • Results

    Weight-sharing and true architecture rankings show Spearman correlations between -0.25 and 0.3 for DARTS, PC-DARTS, GDAS, and Random WS during search.

  • Takeaways & Limitations

    NAS-Bench-1Shot1 enables cheap anytime evaluation and fair comparisons intended to improve reproducibility and reveal one-shot NAS mechanisms.

  • Takeaways & Limitations

    One-shot search typically uses a lower-fidelity proxy model, so the architecture selected there may not be a good architecture in the larger model.

Abstract

from arXiv · show

One-shot neural architecture search (NAS) has played a crucial role in making NAS methods computationally feasible in practice. Nevertheless, there is still a lack of understanding on how these weight-sharing algorithms exactly work due to the many factors controlling the dynamics of the process. In order to allow a scientific study of these components, we introduce a general framework for one-shot NAS that can be instantiated to many recently-introduced variants and introduce a general benchmarking framework that draws on the recent large-scale tabular benchmark NAS-Bench-101 for cheap anytime evaluations of one-shot NAS methods. To showcase the framework, we compare several state-of-the-art one-shot NAS methods, examine how sensitive they are to their hyperparameters and how they can be improved by tuning their hyperparameters, and compare their performance to that of blackbox optimizers for NAS-Bench-101.

1 INTRODUCTION

NAS research has become more efficient through weight sharing, but reproducibility and rigorous evaluation remain difficult. NAS-Bench-1Shot1 addresses these issues with a reusable benchmarking and analysis framework for one-shot NAS.

  • Motivation: Weight sharing reduces NAS search costs by training one large model that subsumes all candidate architectures.This makes search cost comparable to a single function evaluation.
  • Motivation: NAS results are difficult to reproduce because implementations, search spaces, pipelines, hyperparameters, and random seeds vary.The paper motivates a common library of NAS primitives as a step toward more reproducible comparisons.
  • Benchmarking gap: NAS-Bench-101 enables cheap repeated evaluations of discrete NAS optimizers but cannot directly benchmark one-shot NAS optimizers.Its 423k fully evaluated architectures support statistically robust experiments, while its discrete nature creates a compatibility barrier.
  • Contributions: NAS-Bench-1Shot1 reuses NAS-Bench-101 to track architectures found by one-shot methods at every search epoch without retraining them individually.The framework also maps different search-space representations and supports statistically sound analysis.
  • Contributions: The paper introduces a general one-shot NAS framework for fair head-to-head evaluation across recent method variants using one code base.It also provides an open-source implementation intended to facilitate future reproducibility and benchmarking.

2 BACKGROUND AND RELATED WORK

The paper situates one-shot NAS within benchmarked, weight-sharing architecture search and contrasts NAS-Bench-1Shot1 with existing benchmark designs. It emphasizes reusable NAS-Bench-101 evaluations while retaining progressively larger one-shot search spaces.

  • NAS-Bench-101: NAS-Bench-101 exhaustively evaluates constrained cell-based architectures on CIFAR-10, representing each cell as a directed acyclic graph.Its operation set contains 3x3 convolution, 1x1 convolution, and 3x3 max-pool, with limits on nodes and edges.
  • NAS-Bench-101: The constraints and symmetry reduction yield 423k unique architectures, each trained three times and at multiple epoch budgets.The analysis mainly uses results from models trained for 108 epochs.
  • Related benchmarks: NAS-Bench-102 supports weight-sharing evaluation through 15,625 architectures exhaustively evaluated on three image-classification datasets.Both NAS-Bench-102 and this work report architectural overfitting for DARTS across their evaluated datasets.
  • Related benchmarks: NAS-Bench-1Shot1 reuses NAS-Bench-101’s computation to create three one-shot search spaces, including one with 363,648 architectures, without additional computational cost.Its representation mapping changes the one-shot search space while enabling architecture evaluation through NAS-Bench-101.
  • One-shot NAS: Weight sharing improves search efficiency by optimizing architectural and one-shot weights, but search commonly occurs in a lower-fidelity proxy model.The resulting discrete architecture is later trained with more parameters, raising the question of whether proxy-model quality transfers to the larger model.

3 A GENERAL FRAMEWORK FOR BENCHMARKING ONE-SHOT NAS

The framework maps one-shot NAS representations onto NAS-Bench-101 so architectures found during search can be evaluated cheaply throughout the search trajectory. It constructs compatible search spaces, defines architectural choices, and queries benchmark performance from each discrete architecture.

  • Benchmarking procedure: The framework records architecture weights at each epoch, discretizes the corresponding architecture, and queries NAS-Bench-101 for its performance.This mapping enables anytime evaluation despite differences between NAS-Bench-101 and standard one-shot search-space representations.
  • Search-space representation: The one-shot model uses three stacked blocks with three cells per block, while the final evaluation models follow NAS-Bench-101 exactly.The one-shot model uses 16 initial convolution filters rather than the 128 filters used in NAS-Bench-101 to accelerate search.
  • Search-space representation: Cell-level topology is represented as a DAG whose choice blocks select operations and whose architectural weights determine input and output connectivity.The framework introduces edge weights for connections into choice blocks and toward the cell output, while operation weights govern choices within each block.
  • Search-space representation: The framework replaces NAS-Bench-101’s adaptive projections with fixed-channel 1x1 projections because varying tensor dimensions are incompatible with the one-shot model.NAS-Bench-101 cells contain 1x1 projections before every operation, but their adaptive channel counts do not fit the one-shot model.
  • Search-space representation: Three search spaces vary the number of parents per choice block while keeping the total number of parents at 9 to satisfy NAS-Bench-101’s constraint.They contain 6240, 29160, and 363648 architectures with loose ends, respectively.
  • Benchmarking procedure: Evaluation selects the highest-weight operation, chooses top-k parent edges according to each search space, and queries NAS-Bench-101 using the resulting operation list and adjacency matrix.This procedure can produce loose ends when selected nodes do not contribute to the discrete cell output.

4 A GENERAL FRAMEWORK FOR ONE-SHOT NAS METHODS

The framework implements multiple one-shot NAS variants in a shared, modular code basis so their differences can be studied while controlling other factors. It distinguishes continuous-relaxation, path-sampling, and controller-based methods and supports common benchmarking across them.

  • Method variants: PC-DARTS differs from DARTS through partial channel connections, while GDAS replaces Softmax with Gumbel-Softmax and samples single paths during search.These changes are represented as controlled component substitutions relative to DARTS.
  • Method variants: Random WS and ENAS select subnetworks by sampling randomly or from an RNN controller instead of using a continuous relaxation.Both methods train the selected subnetwork within the one-shot model during search.
  • Common implementation: A shared code basis isolates algorithmic differences to a few implementation lines, reducing confounding factors when comparing one-shot NAS variants.The same modular structure also supports incorporating additional methods and identifying which components differ.
  • Benchmarking comparison: Figure 2 compares one-shot optimizers across three NASBench search spaces using anytime test regret and one-shot validation error.Solid lines represent mean ± standard deviation of test regret, while dashed blurred lines represent validation error.
  • Common implementation: The search-space primitives are defined separately from NAS optimizers, allowing either component to be varied while the other remains unchanged.This separation supports studying search-space and optimizer effects in isolation.

5 NAS-BENCH-1SHOT1 AS A BENCHMARK AND ANALYSIS FRAMEWORK

NAS-Bench-1Shot1 evaluates one-shot NAS methods cheaply across search spaces, exposing differences in anytime performance, ranking reliability, hyperparameter robustness, and tunability. The analyses show that validation performance can misrepresent true architecture quality, while careful hyperparameter optimization can substantially improve results.

  • 5.1 COMPARISON OF DIFFERENT ONE-SHOT NAS OPTIMIZERS: NAS-Bench-1Shot1 evaluates five one-shot NAS methods using NAS-Bench-101 queries, reporting mean and standard deviation over six random seeds.The evaluated methods are DARTS, GDAS, PC-DARTS, ENAS, and Random WS; searches ran for 50 epochs under default settings.
  • 5.1 COMPARISON OF DIFFERENT ONE-SHOT NAS OPTIMIZERS: Optimizer rankings differ across search spaces: PC-DARTS performs best in search space 1 but not in the other search spaces.This result shows that optimizer comparisons depend on the search-space configuration.
  • 5.1 COMPARISON OF DIFFERENT ONE-SHOT NAS OPTIMIZERS: GDAS has better anytime performance than the other optimizers across all three benchmarks but prematurely converges in fewer than 5 search epochs.The premature convergence is associated with temperature annealing of the Gumbel Softmax and leads to a sub-optimal local minimum.
  • 5.1 COMPARISON OF DIFFERENT ONE-SHOT NAS OPTIMIZERS: Random WS and ENAS mainly explore poor architectures because their one-shot evaluations correlate weakly with independently measured performance.Their final selections are sampled and ranked using one-shot weights, so weak ranking fidelity affects the selected architecture.
  • 5.2 CORRELATION ANALYSIS: The weight-sharing ranking has almost no correlation with the true ranking for DARTS, PC-DARTS, GDAS, and Random WS, with Spearman coefficients between -0.25 and 0.3.Only ENAS shows some correlation in search space 2 and anticorrelation in search spaces 1 and 3; the analysis covers 137406 architectures in search space 3.
  • 5.3 ROBUSTNESS OF ONE-SHOT NAS OPTIMIZERS: A good hyperparameter setting for one optimizer is not necessarily good for another: at L2 = 27 · 10^-4, GDAS and PC-DARTS perform best while DARTS overfits.Overfitting can also occur for GDAS and PC-DARTS, not only DARTS.
  • 5.4 TUNABILITY OF ONE-SHOT NAS HYPERPARAMETERS: Across all search spaces, the best BOHB configurations outperform default DARTS configurations by up to a factor of 10.The robust configurations typically also outperform state-of-the-art discrete NAS optimizers, while the best incumbents usually come from the lowest 25-epoch budget.

6 CONCLUSION AND FUTURE DIRECTIONS

The paper proposes NAS-Bench-1Shot1 as three benchmarks for cheap, trajectory-level evaluation of one-shot NAS. It is intended to support fairer comparisons, reproducibility, and analysis of how hyperparameters affect these methods.

  • 6 CONCLUSION AND FUTURE DIRECTIONS: NAS-Bench-1Shot1 provides three benchmarks that cheaply track the trajectory and performance of architectures found by one-shot NAS.The framework reuses NAS-Bench-101 evaluations for this analysis.
  • 6 CONCLUSION AND FUTURE DIRECTIONS: The framework compares state-of-the-art one-shot methods and examines their robustness to different hyperparameters.It is designed to study both method performance and sensitivity to search choices.
  • 6 CONCLUSION AND FUTURE DIRECTIONS: The authors intend the framework to facilitate evaluation, improve reproducibility, and provide insights into the mechanisms of one-shot NAS.These are stated future uses of the proposed benchmarks and analysis framework.
  • 6 CONCLUSION AND FUTURE DIRECTIONS: The benchmark search spaces differ in size and connectivity, with search space 1 smallest and search space 3 largest.Search space 3 uses all available intermediate nodes and has the fewest parents per node on average.

B OPTIMIZERS

The benchmark evaluates several one-shot NAS optimizers, differing in how they sample or weight architectures and how they manage computational overhead.

  • DARTS learns operation weights through a weighted continuous relaxation and bilevel optimization using separate training and validation splits.
  • GDAS modifies DARTS by differentiably sampling individual paths with Gumbel-Softmax, reducing memory overhead by evaluating only sampled paths.
  • PC-DARTS reduces memory overhead by evaluating only a random fraction of channels and adds edge normalization to its architectural parameters.
  • Random WS samples architectures per training mini-batch, trains only the selected subnetwork, and selects candidates through one-shot validation before full evaluation.
  • ENAS samples subnetworks with an RNN controller updated by REINFORCE using validation error as reward, while the experiments use fixed batch-size and adapted DARTS hyperparameters.

D COMPARISON OF OPTIMIZERS OVER DIFFERENT BUDGETS

The benchmark compares one-shot NAS optimizers across three NAS-Bench-101-derived search spaces and examines DARTS variants over different epoch budgets.

  • Figure 7 compares different one-shot NAS optimizers on three NAS-Bench-101 search spaces over 100 epochs.
  • Figure 8 compares first- and second-order DARTS on the three search spaces over 200 epochs.

E.1 CUTOUT

Cutout generally reduces the quality of architectures found by one-shot NAS optimizers, with the comparison shown across multiple search spaces and methods.

  • Cutout has a generally negative effect on the quality of solutions found by one-shot NAS optimizers in search space 3.
  • The same negative effect of Cutout also holds for search spaces 1 and 2, as shown for GDAS and PC-DARTS.

E.2 L2 REGULARIZATION

The study examines L2 regularization and broader hyperparameter configurations, finding method-dependent effects on architectures discovered by GDAS, PC-DARTS, and DARTS.

  • Increasing L2 regularization improves architectures found by GDAS and PC-DARTS in search spaces 1 and 2, but degrades DARTS beyond its default.
  • BOHB combines Bayesian Optimization with Hyperband and SuccessiveHalving to evaluate configurations using progressively larger budgets.
  • The simulated wall-clock time combines each algorithm's search time with 108-epoch training time queried from NAS-Bench-101.
  • The experiments define three configuration spaces that progressively add learning rate, momentum, Cutout, batch size, gradient clipping, architectural learning rate, and architectural L2.

F.2 RESULTS

Hyperparameter optimization produced configuration-dependent results: CS2 enabled one-shot NAS optimizers to outperform Regularized Evolution, whereas CS1 and CS3 generally did not. Transferability across search spaces was unclear, while fANOVA indicated cross-space correlation for Cutout and L2 effects.

  • F.2 RESULTS: Optimizing on CS2 produced one-shot NAS configurations that outperformed Regularized Evolution, with learning-rate inclusion crucial to this performance.Under CS1, no sampled configuration outperformed RE; under CS3, only PC-DARTS on search space 3 exceeded the discrete NAS optimizers.
  • F.2 RESULTS: The best configuration found on search space 3 with a 50-epoch BOHB budget showed no clear transferability pattern when applied to search spaces 1 and 2.Results summarize six independent search runs for each optimizer and hyperparameter setting.
  • F.2 RESULTS: fANOVA estimates hyperparameter importance by marginalizing performance over other values using a random-forest model trained on BOHB configurations.The analysis evaluates the relative importance of hyperparameters within specific optimization budgets.
  • F.2 RESULTS: Cutout and L2 effects showed some correlation across search spaces when optimizing CS2 with first-order DARTS.This correlation is presented as a possible basis for studying hyperparameter transfer across search spaces.

H CORRELATION BETWEEN THE ARCHITECTURE SEARCH MODEL AND THE ARCHITECTURE EVALUATION MODEL

The correlation between proxy-model validation error and full architecture evaluation depends on the proxy model’s cell and channel configuration. Increasing channels helped for 9-cell proxies up to a point, whereas 2 initial channels performed best anytime for 3-cell proxies.

  • H CORRELATION BETWEEN THE ARCHITECTURE SEARCH MODEL AND THE ARCHITECTURE EVALUATION MODEL: The experiment sampled 150 architectures from search space 3 and compared proxy-model validation error with full NAS-Bench-101 evaluation using Spearman rank correlation.Proxy models varied in total cells and initial channels.
  • H CORRELATION BETWEEN THE ARCHITECTURE SEARCH MODEL AND THE ARCHITECTURE EVALUATION MODEL: For 3-cell proxies, 2 initial channels produced the strongest anytime correlation, while additional channels worsened early performance without improving the final result.The findings suggest that effective cell-channel combinations depend on the proxy configuration.
Loading 2001.10422v2…