PaperlayerView exact revision on arXiv

Source-linked AI summary

NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies

Arjun Krishnakumar, Colin White, Arber Zela, Renbo Tu, Mahmoud Safari, Frank Hutter

arXiv:2210.03230v1cs.LGcs.AIstat.ML

TL;DR

Zero-cost proxies can estimate architecture performance quickly, but prior work left their cross-task generalizability and complementarity under-studied. NAS-Bench-Suite-Zero evaluates 13 proxies across 28 tasks in a unified framework, and analyses show complementary information and improved NAS surrogate predictions. The work reports that combining proxies improves surrogate and NAS performance, while its analysis remains empirical and incomplete for some benchmark spaces.

  • Problem

    Prior studies found that zero-cost proxies often generalize poorly across benchmarks, and it remained unclear whether they could be combined effectively or integrated into NAS algorithms.

    Source: §1 · pp. 1–2

  • Method

    The authors build a unified suite with precomputed evaluations of 13 proxies across 28 tasks and use it to analyze proxy behavior and integration into NAS.

    Source: §1 · p. 2, §3 · p. 4, §6 · p. 9

  • Results

    Combining proxies increases the Spearman rank correlation of surrogate predictions by up to 42% and improves predictor-based NAS algorithms.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Precomputed proxy scores reduce experiment runtimes and support comparisons across methods and diverse tasks.

    Source: §6 · p. 10

    The study is empirical, and NAS-Bench-301 proxy evaluations cover only about 11,000 sampled architectures rather than the full space.

    Source: §6 · p. 10

Abstract

from arXiv · show

Zero-cost proxies (ZC proxies) are a recent architecture performance prediction technique aiming to significantly speed up algorithms for neural architecture search (NAS). Recent work has shown that these techniques show great promise, but certain aspects, such as evaluating and exploiting their complementary strengths, are under-studied. In this work, we create NAS-Bench-Suite: we evaluate 13 ZC proxies across 28 tasks, creating by far the largest dataset (and unified codebase) for ZC proxies, enabling orders-of-magnitude faster experiments on ZC proxies, while avoiding confounding factors stemming from different implementations. To demonstrate the usefulness of NAS-Bench-Suite, we run a large-scale analysis of ZC proxies, including a bias analysis, and the first information-theoretic analysis which concludes that ZC proxies capture substantial complementary information. Motivated by these findings, we present a procedure to improve the performance of ZC proxies by reducing biases such as cell size, and we also show that incorporating all 13 ZC proxies into the surrogate models used by NAS algorithms can improve their predictive performance by up to 42%. Our code and datasets are available at https://github.com/automl/naslib/tree/zerocost.

1 Introduction

Zero-cost proxies promise fast architecture-performance estimates, but prior studies found weak cross-benchmark generalization and left their complementarity and integration into NAS largely unresolved. NAS-Bench-Suite-Zero unifies proxy evaluation across tasks and supports analyses showing complementary information and improved surrogate predictions.

1 Introduction

  • 13 zero-cost proxies across 28 tasks are collected in a unified, extensible framework with precomputed scores for faster experiments.The framework also provides scripts for large-scale studies and avoids confounding factors from different implementations.Source: §1 · p. 2
  • Combining all 13 proxies raises the Spearman rank correlation of surrogate predictions by up to 42% and improves BANANAS and NPENAS performance.The analysis finds substantial complementary information among proxies conditioned on validation accuracy.Source: §1 · p. 2
  • The analysis covers proxy generalizability, biases, and mutual information, including whether proxies provide complementary information.The paper describes this as the first information-theoretic analysis of ZC proxies.Source: §1 · p. 2
  • A procedure to reduce biases such as favoring larger architectures or architectures with more convolution operations improves proxy performance.The authors note this may have consequences for future proxy design.Source: §1 · p. 2

2 Background and Related Work

NAS automates architecture selection, and zero-cost proxies offer a fast way to estimate relative performance. Prior studies covered limited task counts and proxy combinations, motivating broader, publicly available evaluation and more comprehensive use of complementary proxy information.

Background

  • NAS seeks architectures with high validation accuracy or favorable application-specific trade-offs, while performance predictors estimate architecture performance without full training.Zero-cost proxies require a single forward pass and often take about five seconds.Source: §2 · p. 3

Zero-cost proxies

  • Zero-cost proxies include data-independent methods, pruning-at-initialization methods, and methods inspired by neural tangent kernels.The original proxy estimated data separability into linear regions of the output space [22].Source: §2 · p. 3

Search spaces and tasks

  • The benchmark landscape spans cell-based and macro search spaces, multiple datasets, and diverse tasks, including additions from NAS-Bench-360 and Taskonomy.NAS-Bench-101, NAS-Bench-201, NAS-Bench-301, and TransNAS-Bench-101 are among the included benchmark families.Source: §2 · p. 3

Large-scale studies of ZC proxies

  • Earlier large-scale studies evaluated at most 12 tasks and did not release proxy values publicly, while prior proxy-combination work used voting with at most four proxies [1].OMNI used one proxy, and ProxyBO dynamically selected one proxy per iteration [32].Source: §2 · pp. 3–4
  • NAS-Bench-Suite-Zero publicly releases proxy values, combines proxies nontrivially, and uses 13 proxies simultaneously to exploit complementary information.This contrasts with prior voting approaches that combined at most four proxies [1].Source: §2 · p. 4

3 Overview of NAS-Bench-Suite-Zero

NAS-Bench-Suite-Zero provides unified proxy implementations and precomputed scores across benchmark tasks, enabling comparisons and NAS experiments with substantially reduced evaluation time. Its coverage and sampling vary by benchmark, and the section reports large runtime savings from querying precomputed scores.

3 Overview of NAS-Bench-Suite-Zero

  • The suite extends NAS-Bench-Suite with 28 benchmarks and uses one framework to compare proxies without implementation, software-version, or training-pipeline confounds.NAS-Bench-301 uses 11,221 sampled architectures; NAS-Bench-101 uses 10,000, while several other tasks use all architectures or 200 trained architectures.Source: §3 · p. 4
  • Figure 2 presents Spearman rank correlations between proxy values and validation accuracy for each proxy and benchmark.Its rows and columns are ordered by mean scores across columns and rows, respectively.Source: Figure 2 · p. 5

Speedups and recommended usage

  • Querying the NAS-Bench-Suite-Zero API takes 10^-5 seconds, compared with an average proxy computation time of 2.6 seconds and a maximum of 205 seconds.The maximum is for grasp on TransNAS-Bench-101-Macro Autoencoder.Source: §3 · p. 5
  • 100-iteration proxy-based NAS runs with 100 candidates per iteration take roughly 9 hours to evaluate proxies, but run in under one minute on NAS-Bench-Suite-Zero.Across the paper's experiments, the suite reduces computation time by at least three orders of magnitude on queryable benchmarks.Source: §3 · p. 5
  • The authors recommend extensive trials, ablations, and proxy coverage, while adding proxy-evaluation time when reporting real-world NAS runtime.The advice follows the stated reduction in experiment runtime by at least three orders of magnitude on queryable benchmarks.Source: §3 · p. 5

4 Generalizability, Mutual Information, and Bias of ZC Proxies

The section evaluates how ZC proxies generalize across benchmarks, complement one another in explaining validation accuracy, and exhibit biases that may be mitigated. It finds benchmark-dependent performance and complementarity, alongside bias reduction that can improve proxy performance.

4.1 RQ 1: How well do ZC proxies generalize across different benchmarks?

  • Only a few ZC proxies generalize well across most benchmarks, although proxy scores can help assess benchmark similarities.nwot and flops have the highest rank correlations across benchmarks; similar search spaces show high score correlation, including Trans-NAS-Bench-101-Macro and NAS-Bench-201 clusters.Source: §4.1 · pp. 5–6, Figure 3 · p. 6

4.2 RQ 2: Are ZC proxies complementary with respect to explaining validation accuracy?

  • ZC proxies provide substantial complementary information on some benchmarks, but the degree of complementarity depends heavily on the benchmark.Adding proxies yields substantial information gain even under random ordering; greedy ordering is not far from the optimum, and synflow and plain together provide the most information in the reported analysis.Source: §4.2 · pp. 6–7

4.3 RQ 3: Do ZC proxies contain biases, such as a bias toward certain operations or sizes, and can we mitigate these biases?

  • Biases include correlations with cell size and operation mix, and the bias of a proxy can differ across search spaces.Synflow’s high cell-size correlation can favor larger architectures; completely removing bias may hurt performance because validation accuracy itself can correlate with bias measures.Source: §4.3 · pp. 7–8
  • Bias-mitigation strategies can improve ZC proxy performance, and many proxies exhibit biases whose patterns vary across search spaces.The performance strategy improves performance; equalize sometimes performs on par when the ground truth’s correlation with the bias metric is known.Source: §4.3 · p. 8

5 Integration into NAS

The paper adds all 13 ZC proxies as features to predictor-based NAS models and evaluates their predictive and search performance. On NAS-Bench-201 CIFAR-100, adding the proxies substantially improves standalone predictor correlation, and they also boost NAS performance, especially early in search.

5 Integration into NAS

  • The integration adds all 13 ZC proxies directly as features to the surrogate models used by predictor-based NAS algorithms.The experiments use BANANAS and NPENAS.Source: §5 · pp. 8–9
  • 41.7% improvement in Spearman rank correlation, from 0.640 ± 0.0420 to 0.908 ± 0.012, when ZC proxies are added to XGBoost on NAS-Bench-201 CIFAR-100.The experiment averaged 100 trials; ZC proxies alone achieved 0.907 ± 0.013.Source: §5 · p. 9
  • Adding ZC proxies as predictor features gives BANANAS and NPENAS a performance boost, especially in the early stages of search.Each algorithm received 200 architecture evaluations, with performance averaged over 400 trials.Source: §5 · pp. 8–9

6 Conclusions, Limitations, and Broader Impact

NAS-Bench-Suite-Zero releases precomputed scores for 13 proxies across 28 tasks and supports large-scale analysis of their complementary information. Combining proxies improves surrogate and NAS performance, while the analysis remains empirical and incomplete for some benchmarks.

Conclusions

  • The released dataset contains 1.5 million precomputed proxy evaluations across 28 tasks, reducing the overhead of comparing methods across diverse tasks.The paper gives an example of reducing an experiment from 9 hours to 4 minutes.Source: §6 · pp. 9–10
  • The analysis finds substantial complementary information among proxies, and using several together improves both surrogate models and NAS algorithms.The authors present this as motivation for ensembling all 13 proxies in predictor-based NAS.Source: §6 · p. 10

Limitations

  • The analysis is empirical, and NAS-Bench-301 proxy evaluation covers only 11,000 architectures because evaluating its 10^18-architecture space is computationally infeasible.The authors suggest training a surrogate model to predict proxy performance on the remaining architectures.Source: §6 · p. 10

Broader Impact

  • Precomputing proxy scores lets researchers run NAS trials cheaply on CPUs, which the authors say can reduce experiment-related carbon emissions.They cite the high GPU consumption of prior NAS research as a reason this reduction is worthwhile.Source: §6 · p. 10

A Dataset Documentation

The appendix documents the dataset and repository, including its license, maintenance plans, conduct policy, and datasheet. It also situates the work among prior proxy studies and highlights its broader scale and use of 13 proxies.

A Dataset Documentation

  • The documentation section directs readers to the repository for dataset details, links, usage information, and tutorials.Source: A Dataset Documentation · p. 15

A.1 Author responsibility and license

  • The repository uses the Apache License 2.0, and the authors also provide licenses for the datasets they used.Source: §A.1 · p. 15

A.2 Maintenance plan

  • The authors plan to maintain the repository actively and welcome community contributions.Source: §A.2 · p. 15

A.3 Code of conduct

  • The project adopts Contributor Covenant version 2.0 as its code of conduct.Source: §A.3 · p. 15

A.4 Datasheet

  • The paper includes a datasheet for NAS-Bench-Suite-Zero.Source: §A.4 · p. 15

B Related Work Continued

  • Prior studies evaluate fewer tasks or proxies, and earlier combinations used voting among three or four proxies; this work combines 13 nontrivially and studies the largest set to date.Prior work also examined proxy performance across benchmarks and predictor-based NAS settings.Source: B Related Work Continued · pp. 15–16

B.1 Theoretical results for ZC proxies

The appendix reviews theoretical analyses of proxy biases, relationships among proxy values, generalization bounds, and theory-inspired proxy designs. It frames further theory as important for understanding robustness and designing higher-performing proxies.

B.1 Theoretical results for ZC proxies

  • Synflow theoretically favors larger architectures: adding a fully connected layer to an MLP increases its value, and the authors say the intuition should extend to CNNs.The paper notes its empirical results confirm the cited theoretical finding.Source: §B.1 · p. 16
  • Under stated Lipschitz assumptions, grad_norm, snip, and grasp are asymptotically similar up to constants to the NTK trace norm at initialization, implying high correlation.Source: §B.1 · p. 16
  • A generalization bound expresses DNN error as training error plus O(κ/M), with M set to grad_norm, snip, or grasp and κ the NTK condition number.A corollary also bounds generalization error using proxy values and other fixed network constants, without the training-error term.Source: §B.1 · p. 16
  • Theory-inspired proxies include NASI, based on an NTK trace-norm analysis, TE-NAS using the NTK spectrum and linear-region count, and NNGP-NAS using Monte Carlo approximation.Source: §B.1 · p. 16
  • The authors identify further theory as important for understanding proxy robustness across datasets and designing higher-performing proxies.Source: §B.1 · p. 16

C.1 Details from Section 4.1: generalization

The appendix extends the generalization, information-theoretic, bias, runtime, and predictor analyses. It reports gains from adding proxy features, while also examining alternative ranking metrics, a larger search space, discretization choices, and feature importance.

C.1 Details from Section 4.1: generalization

  • Precision@K and BestRanking@K show trends similar to Spearman, and may better match NAS's goal of finding the very best architectures.The appendix recomputes the generalization results at K = 5, 25, and 100.Source: §C.1 · p. 17

C.1.1 Initial results with FBNet

  • On FBNet, snip, synflow, and flops perform surprisingly well among proxies, with flops highest; Table 6 reports Spearman correlations for 100 randomly drawn architectures.FBNet has a search space of 10^21 architectures.Source: §C.1.1 · p. 17, Table 6 · p. 17

C.2 Details from Section 4.2: information theory

  • Conditional entropy requires discretized scores, so the analysis bins values using Sturges' rule; information gain is order-dependent and measures one proxy's complementary information given another.With strategic proxy selection, marginal information gain usually drops substantially around four proxies; greedy selection often matches brute force, while random selection can be unstable.Source: §C.2 · pp. 17–18

C.3 Details from Section 4.3: biases

  • Table 7 reports the highest and lowest absolute correlations for each bias metric and task; for NB201-CF10 Conv:pool bias, synflow correlates at 0.76 and grasp at -0.01.Params and flops are excluded from parameter-count bias because their correlation is trivially 1.00; operation biases are unavailable for TransNAS-Bench-101-Macro.Source: §C.3 · p. 18, Table 7 · p. 19

D Details from Section 5

  • 0.640 to 0.908: adding 13 ZC proxies raises XGBoost Spearman rank correlation on NAS-Bench-201 CIFAR-100 with 100 training architectures, averaged over 100 trials.The authors report consistent improvements across benchmarks; adding proxies also improves predictions when 1,000 architectures train XGBoost.Source: D Details from Section 5 · p. 19, Table 9 · p. 20

D.1 Feature importances of ZC proxies

  • Feature importance is measured by how often each feature splits data across XGBoost trees, with results shown for training sets of 100 and 1,000 architectures.The appendix also gives Table 9's average Spearman correlations for models using encoding alone, proxy features alone, or both.Source: §D.1 · pp. 19–20, Table 9 · p. 20

C.4 NAS-Bench-Suite-Zero Speedup Details

  • The appendix compares runtimes for each experiment type with and without NAS-Bench-Suite-Zero, measured on an Intel Xeon Gold 6242 CPU.The main text reports speedups of orders of magnitude for NAS experiments previously run with NAS-Bench-Suite.Source: §C.4 · p. 18, Table 8 · p. 20

D.2 Ablation study on the number of ZC proxies

Using a greedy proxy ordering, the ablation finds the best performance with all 13 proxies, though gains after 6–8 proxies are small.

D.2 Ablation study on the number of ZC proxies

  • All 13 ZC proxies achieve the best performance on every task, with ties in some cases.The XGBoost surrogate was trained on 1000 randomly drawn architectures, and proxy order was computed greedily.Source: §D.2 · p. 20
  • Adding proxies beyond 6–8 yields only a small improvement up to the full set of 13.This pattern is consistent with the mutual-information study in Section 4.3.Source: §D.2 · p. 20

D.3 Additional NAS results

Additional NAS experiments report improved performance when ZC proxy scores are added to surrogate models. The experiments cover BANANAS on 11 benchmarks and NPENAS on complete benchmarks only.

D.3 Additional NAS results

  • ZC proxy scores improve BANANAS performance across all 11 tested benchmarks.Figure 18 compares BANANAS with vanilla XGBoost against XGBoost using proxy scores concatenated to the architecture encoding.Source: §D.3 · p. 20
  • NPENAS is tested only on NAS-Bench-201 and TransNAS-Bench-101 because its mutation step requires complete benchmarks.Figure 19 compares NPENAS with vanilla XGBoost against XGBoost augmented with ZC proxy scores.Source: §D.3 · p. 20
  • Table 10 reports average Spearman rank correlations between XGBoost predictions and validation accuracy for encoding-only, ZC-only, and combined features.The comparison uses 1000 models to train XGBoost.Source: Table 10 · p. 21

E ZC Proxy Competition

The Zero Cost NAS Competition evaluated proxies by Kendall-Tau ranking on specified development and test tasks. Its winning teams used normalized proxy variants and a proxy product.

E ZC Proxy Competition

  • The winning teams used normalized synflow, normalized fisher, and a product of grad_norm and params.Kendall-Tau rank correlation, averaged across three benchmarks, scored rankings in the test phase.Source: E ZC Proxy Competition · p. 21
  • The final test phase covered NB101 with CIFAR10, NB201 with ImageNet16x120, NB301 with CIFAR10, and two TransNAS-Bench-101 tasks.The TransNAS-Bench-101 tasks were Macro with Object Classification and Micro with Object Classification.Source: E ZC Proxy Competition · p. 21
  • The development phase covered NB201, NB301, and TNB101-Micro with Ninapro, SVHN, and Spherical-CIFAR100 task combinations.The passage lists Ninapro and SVHN for NB201 and NB301, and adds Spherical-CIFAR100 for TNB101-Micro.Source: E ZC Proxy Competition · p. 21
  • Figure 6 displays Spearman rank correlations between proxy values and validation accuracy across proxies and benchmarks.Rows and columns are ordered by their mean scores.Source: Figure 6 · p. 22
  • Figures 8 and 9 show Precision@K and BestRanking@K, respectively, between proxy values and validation accuracy across proxies and benchmarks.Rows and columns are ordered by mean scores.Source: Figure 8 · p. 24, Figure 9 · p. 25
  • Figures 10 and 13 show Pearson correlations across proxy pairs and benchmark pairs, respectively.The entries are ordered by mean scores across rows and columns.Source: Figure 7 · p. 23, Figure 10 · p. 26
  • Figures 11–16 plot conditional entropy and information gain for proxy pairs, and conditional entropy against proxy-set size under three orderings.Figure 16 notes overlap between the minimum k-tuple and greedy orderings in the first two plots.Source: Figure 11 · p. 27, Figure 12 · p. 28, Figure 13 · p. 29
  • Figures 17–21 cover proxy-count ablation, BANANAS and NPENAS performance comparisons, and XGBoost feature importance at two training-set sizes.The feature-importance figures use sets of 100 and 1000 architectures.Source: Figure 17 · p. 33, Figure 18 · p. 34, Figure 19 · p. 35

Related papers

Loading 2210.03230v1…