PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Cost Proxies for Lightweight NAS

Mohamed S. Abdelfattah, Abhinav Mehrotra, Łukasz Dudziak, Nicholas D. Lane

arXiv:2101.08134v2cs.LGcs.AIcs.NE

TL;DR

NAS requires evaluating many fully trained candidate networks, making search computationally expensive. This paper introduces single-minibatch zero-cost proxies based mainly on pruning-at-initialization metrics and uses them within existing search algorithms. Across datasets and search methods, the proxies improve ranking or search efficiency, including a reported 4× speedup on NAS-Bench-101.

  • Problem

    NAS candidate evaluation is costly because conventional searches may require fully training many neural networks, while proxy rankings must preserve final-accuracy ordering.

  • Method

    The paper adapts pruning-at-initialization metrics to score whole networks from a single minibatch and integrates them into multiple NAS search algorithms.

  • Results

    Across four search algorithms and three NAS datasets, zero-cost integration significantly improves search speed, including up to 4× on NAS-Bench-101.

  • Takeaways & Limitations

    Zero-cost proxies can improve NAS sample efficiency while using substantially less computation than conventional reduced-training proxies.

  • Takeaways & Limitations

    A reduced-training proxy that works well in one search space may not work well in another, even for CIFAR-10 image classification.

Abstract

from arXiv · show

Neural Architecture Search (NAS) is quickly becoming the standard methodology to design neural network models. However, NAS is typically compute-intensive because multiple models need to be evaluated before choosing the best one. To reduce the computational power and time needed, a proxy task is often used for evaluating each model instead of full training. In this paper, we evaluate conventional reduced-training proxies and quantify how well they preserve ranking between multiple models during search when compared with the rankings produced by final trained accuracy. We propose a series of zero-cost proxies, based on recent pruning literature, that use just a single minibatch of training data to compute a model's score. Our zero-cost proxies use 3 orders of magnitude less computation but can match and even outperform conventional proxies. For example, Spearman's rank correlation coefficient between final validation accuracy and our best zero-cost proxy on NAS-Bench-201 is 0.82, compared to 0.61 for EcoNAS (a recently proposed reduced-training proxy). Finally, we use these zero-cost proxies to enhance existing NAS search algorithms such as random search, reinforcement learning, evolutionary search and predictor-based search. For all search methodologies and across three different NAS datasets, we are able to significantly improve sample efficiency, and thereby decrease computation, by using our zero-cost proxies. For example on NAS-Bench-101, we achieved the same accuracy 4$\times$ quicker than the best previous result. Our code is made public at: https://github.com/mohsaied/zero-cost-nas.

1 INTRODUCTION

NAS automates neural-network design but makes each candidate evaluation costly because full training can take hours or days. The paper addresses this bottleneck with single-minibatch proxies and integrates them into several search algorithms.

  • Full training can take hours or days for each neural-network evaluation in conventional NAS.
  • The paper proposes scoring networks with a single minibatch and one forward/backward propagation pass.
  • Zero-cost proxies adapt pruning-at-initialization metrics to rank complete network candidates rather than individual parameters.
  • The study compares zero-cost proxies with reduced-computation training proxies using rank consistency across large-scale datasets.
  • Ablations evaluate the zero-cost proxies across five NAS benchmarks and varied settings.
  • Zero-cost metrics are integrated with random search, reinforcement learning, aging evolution, and predictor-based search, producing speedups up to 4× on NAS-Bench-101.

2 RELATED WORK

Prior work reduces NAS cost through training or architecture simplification, while pruning-at-initialization methods provide untrained saliency signals that can also support architecture ranking.

  • Weight sharing and smaller datasets reduce candidate-training costs during NAS evaluation.
  • EcoNAS studies reduced-training proxies by shrinking input size, model size, training samples, or epochs.
  • Pruning methods rank parameters by saliency, but traditional criteria require training before computation.
  • Pruning-at-initialization methods compute saliency without training, motivating their adaptation for NAS.
  • Prior NAS work used pruning-related scores and Jacobian covariance to rank architectures without training.

3 PROXIES FOR NEURAL NETWORK ACCURACY

The paper reviews reduced-training accuracy proxies and adapts pruning-at-initialization saliency metrics to score whole networks, alongside Jacobian covariance based on minibatch activations.

  • Conventional NAS proxies: Conventional NAS proxies reduce epochs, samples, input resolution, or model size to predict final accuracy.
  • Conventional NAS proxies: EcoNAS reported a proxy using approximately 1/4 of model size and input resolution, all training samples, and 1/10 of epochs.
  • Zero-cost proxies: Grad norm scores a network by summing gradient Euclidean norms after one minibatch of training data.
  • Zero-cost proxies: The paper adapts parameter-pruning saliency metrics to score and rank entire neural-network models.
  • Pruning-based metrics: SNIP, GRASP, and synaptic-flow methods estimate pruning-related effects at initialization using a single minibatch or related saliency formulation.
  • Fisher: The Fisher metric is aggregated from activation-level saliencies to quantify the importance of network components.
  • Jacobian covariance: Jacob cov measures activation correlation across different minibatch inputs, with lower correlation expected to indicate better input differentiation.

4 EMPIRICAL EVALUATION OF PROXY TASKS

The evaluation compares conventional reduced-training proxies with zero-cost metrics across NAS benchmarks and model collections. Zero-cost metrics use initialization-time gradient computations and generally provide stronger or more consistent ranking correlations at substantially lower computational cost.

  • Evaluation criteria: Spearman ρ measures how well each proxy ranks models relative to ground-truth accuracy rankings.The evaluation focuses on empirical ranking quality rather than explaining why individual metrics work.
  • NAS-Bench-201: 15,625 NAS-Bench-201 models are used to compare EcoNAS proxies with zero-cost proxies across three image-classification datasets.NAS-Bench-201 provides cell-based CNN models with corresponding training statistics.
  • Conventional proxies: A 50× gap separates EcoNAS theoretical FLOPS speedup from measured GPU speedup for the r8c4e20 configuration.The proxy matches approximately 1/10 of a full-training epoch in FLOPS but takes time equivalent to five full-training epochs on the tested GPU.
  • Zero-cost proxies: Zero-cost proxies compute scores from gradient calculations on a single initialization-time minibatch, requiring only a forward/backward pass.The combined snip metric uses three minibatches and outperforms individual metrics, while synflow is strongest across the three NAS-Bench-201 datasets.
  • Other search spaces: Synflow is the only consistent metric across the analyzed alternative NAS benchmarks, but its rank correlation is about 0.3 there versus about 0.8 on NAS-Bench-201.The alternative benchmarks include larger search spaces and different tasks, including NLP and speech recognition.
  • Integration with NAS: Zero-cost metrics improve sample efficiency for random search, reinforcement learning, aging evolution, and predictor-based search, with speedups up to 4× on NAS-Bench-101.The integration is evaluated across three NAS datasets.

5 ZERO-COST NAS

Zero-cost proxies are integrated into NAS search through warmup and move-proposal strategies, improving search efficiency across multiple algorithms and benchmarks.

  • Zero-Cost Warmup: Zero-cost warmup scores many untrained models to initialize NAS algorithms, exploiting T ≪ N when more models can be scored than trained.The approach is applied to random search, reinforcement learning, aging evolution, and predictor-based search.
  • Zero-Cost Move Proposal: Zero-cost move proposal evaluates local candidate mutations with proxy scores before selecting models for training.For aging evolution, all edit-distance-1 mutations are considered and the highest-scoring candidate is selected.
  • NAS-Bench-201: 5.6× and 2.3× speedups let aging evolution and binary predictor search reach 73.5% accuracy faster on NAS-Bench-201.Search speed improves across all four evaluated search types, while reinforcement learning degrades with 15k warmup points.
  • NAS-Bench-101: Even with global/top-10% rank correlations of 0.37/0.14, zero-cost methods substantially improve NAS-Bench-101 search and set a new state-of-the-art result.The binary predictor shows no improvement or degradation on this benchmark.
  • NAS-Bench-ASR: 68, 173 and 87 trained models replace more than 1000, more than 1000 and 138 models, respectively, to reach 21.3% PER on NAS-Bench-ASR.These reductions apply to RAND, RL and AE after 2000 models of zero-cost warmup.

6 DISCUSSION

The discussion attributes zero-cost NAS gains to better initialization and local mutation selection, showing how synflow increases access to accurate models.

  • Warmup: Synflow warmup places more top-5%-accurate models among the top 64 candidates than random warmup.Random warmup returns about 3 of 64 top-5% models, while synflow returns a higher number.
  • Warmup: Initializing aging evolution with more accurate models makes beneficial mutations more likely and can accelerate discovery of top models.The paper notes that this effect can occur even when global or local correlation is low, including on NAS-Bench-ASR.
  • Move Proposal: Synflow raises the probability of selecting the best edit-distance-1 mutation from about 4% to over 30% on NAS-Bench-201 and to 12% on NAS-Bench-101.On NAS-Bench-ASR, the probability increases from 7.7% to 10%.

7 CONCLUSION

The paper concludes that synflow is a robust zero-cost proxy for accelerating NAS across search algorithms and benchmarks.

  • Contributions: Six zero-cost proxies are introduced, but only synflow remains robust across all evaluated datasets for global and top-10% rank correlation.The proxies are mainly based on pruning-at-initialization methods and rank neural network models for NAS.
  • Contributions: Zero-cost warmup and move proposal produce significant speedups across four search algorithms and three NAS benchmarks.The methods set new state-of-the-art results on NAS-Bench-101 and NAS-Bench-201.

A APPENDIX

The appendix contains additional empirical results supporting the paper’s main arguments and hypotheses.

  • The appendix lists many results beyond those presented in the main text because the study is empirically driven.

A.1 EXPERIMENTAL DETAILS

The EcoNAS proxy experiments use a shortened 40-epoch learning-rate annealing schedule, while NAS benchmark searches query random stored seeds to mimic uncached evaluation.

  • EcoNAS proxies anneal the learning rate to zero over 40 epochs instead of the standard 200-epoch NAS-Bench-201 schedule.The reduced schedule is described as a common technique for speeding up proxy convergence.
  • The EcoNAS hyper-parameters used for NAS-Bench-201 are listed in Table 6.
  • NAS benchmark experiments query a random result seed whenever a model is “trained,” mimicking a real NAS pipeline without caching.The datasets provide multiple result seeds for each model.

A.2 GPU RUNTIME FOR ECONAS

EcoNAS proxies reduce computation substantially, but their measured real-speedup gains show diminishing returns as batch size increases.

  • 64× less computation for r8c4 than r32c16 yields a maximum 4× real speedup, even when batch size increases.
  • Higher batch sizes for EcoNAS proxies have diminishing returns in measured speedup on 10 randomly sampled NAS-Bench-201 CIFAR-10 models.

A.3 TABULATED RESULTS

The tabulated analyses compare search outcomes, ranking quality, local-neighborhood behavior, and sensitivity across datasets, algorithms, and zero-cost metrics. Synflow is repeatedly identified as the strongest and most consistent metric, while aging evolution remains robust to weaker metrics.

  • Tabulated search results: Zero-cost NAS comparisons report accuracy or PER after 50 trained models and the number of models needed to reach specified thresholds on NAS-Bench-201 and NAS-Bench-ASR.
  • Top-10% model analysis: Synflow is the only zero-cost metric with consistent top-10% rank correlation across different datasets and performs best on average.Table 9 reports Spearman ρ for the top 10% of points across NAS search spaces.
  • Warmup and move proposal: Synflow is the most powerful metric tested for ranking top-5% models within the top 64 returned models.
  • Warmup and move proposal: Grad norm and jacob cov distinguish similar models effectively in local neighborhoods, although synflow remains consistently best overall.
  • Warmup and move proposal: Synflow most often selects the validation-accuracy-best model within local clusters of edit distance 1.The probability is averaged over 1000 randomly sampled clusters and represents the best possible zero-cost move proposal.
  • Search results: On NAS-Bench-101, Figure 7 presents search curves for all considered algorithms, while Figure 1 compares synflow search speedup.
  • Sensitivity analysis: All metrics are relatively unaffected by initialization and minibatch-size variation, except fisher benefits from zero-initialized biases.
  • Search results: Random search benefits greatly from strong metrics but can deteriorate with weaker ones, whereas aging evolution can recover and reach top accuracy in most cases.
Loading 2101.08134v2…