PaperlayerView exact revision on arXiv

Source-linked AI summary

How Powerful are Performance Predictors in Neural Architecture Search?

Colin White, Arber Zela, Binxin Ru, Yang Liu, Frank Hutter

arXiv:2104.01177v2cs.LGcs.NEstat.ML

TL;DR

Performance predictors reduce NAS costs, but methods from different families have not been systematically compared under varying initialization and query-time constraints. This study compares 31 predictors across settings and shows that complementary predictors can be combined for stronger predictive power.

  • Problem

    Different performance-predictor families lack systematic comparison across search spaces, metrics, and initialization- versus query-time constraints.

  • Method

    The study evaluates 31 predictors across four search spaces and combines learning-curve, zero-cost, and model-based predictors in OMNI.

  • Results

    Predictors from different families are complementary, and combining them substantially improves predictive performance.

  • Takeaways & Limitations

    The results provide recommendations for choosing predictors under different runtime constraints and a library for fair future comparisons.

  • Takeaways & Limitations

    The work is one level of abstraction away from real applications.

Abstract

from arXiv · show

Early methods in the rapidly developing field of neural architecture search (NAS) required fully training thousands of neural networks. To reduce this extreme computational cost, dozens of techniques have since been proposed to predict the final performance of neural architectures. Despite the success of such performance prediction methods, it is not well-understood how different families of techniques compare to one another, due to the lack of an agreed-upon evaluation metric and optimization for different constraints on the initialization time and query time. In this work, we give the first large-scale study of performance predictors by analyzing 31 techniques ranging from learning curve extrapolation, to weight-sharing, to supervised learning, to "zero-cost" proxies. We test a number of correlation- and rank-based performance measures in a variety of settings, as well as the ability of each technique to speed up predictor-based NAS frameworks. Our results act as recommendations for the best predictors to use in different settings, and we show that certain families of predictors can be combined to achieve even better predictive power, opening up promising research directions. Our code, featuring a library of 31 performance predictors, is available at https://github.com/automl/naslib.

1 Introduction

NAS performance predictors reduce the need to fully train architectures, but methods from different families had not been systematically compared under varying initialization and query-time constraints. This study compares 31 predictors across settings and shows that complementary families can be combined for stronger predictive power.

  • Early NAS methods fully trained thousands of architectures, whereas newer methods estimate the performance of partially trained or untrained networks.
  • Model-based predictors often require hundreds of fully trained architectures for initialization, while learning-curve methods trade low initialization time for high query time.
  • The study asks how zero-cost, model-based, learning-curve, and weight-sharing predictors compare under different initialization and query-time constraints.
  • The authors compare 31 predictors across four search spaces and four datasets using Pearson, Spearman, Kendall Tau, and sparse Kendall Tau metrics.
  • Predictors from different families are complementary and can be combined to achieve significantly higher performance.
  • The released NASLib-based code supports choosing predictors under runtime constraints and reproducing or extending the comparisons.

2 Related Work

Related work includes broad NAS developments and earlier predictor comparisons, but prior studies were limited in search-space, dataset, or objective coverage. These limitations motivate a larger cross-family evaluation.

  • NAS research has expanded from reinforcement learning and evolutionary search to one-shot and predictor-based algorithms.
  • Earlier predictor comparisons examined selected methods within narrower settings rather than comparing all major predictor families across multiple search spaces.

3 Performance Prediction Methods for NAS

Performance predictors differ in how they allocate initialization and query computation. The main families range from supervised models and learning-curve methods to zero-cost and weight-sharing approaches, each offering distinct runtime tradeoffs.

  • A performance predictor takes an architecture specification as input and predicts its final accuracy or architecture ranking.
  • Each predictor has an initialization routine for general pre-computation and a query routine for architecture-specific prediction.
  • Model-based methods: Model-based methods fully train many architectures to create supervised data, then learn a model mapping architecture encodings to performance.
  • Learning curve-based methods: Learning-curve methods extrapolate final performance from partial training and require little or no initialization time, but typically incur high query time.
  • Hybrid methods: Hybrid methods combine architecture features with partial learning curves and generally have both high initialization and query times.
  • Zero-cost methods: Zero-cost methods compute predictors from a single forward/backward pass on one minibatch, yielding no initialization time and very short queries.
  • Weight sharing methods: Weight sharing trains one supernetwork whose shared weights provide rapid performance estimates for architectures in the search space.
  • Runtime tradeoffs: The preferred predictor family depends on total runtime, initialization budget, query budget, and the number of architectures to evaluate.

4 Experiments

The experiments compare 31 performance predictors across search spaces, datasets, runtime budgets, training/test distributions, and predictor-based NAS frameworks. Predictor performance depends on initialization and query-time constraints, while combining complementary predictor families can improve results across settings.

  • Experimental setup: The study evaluates predictor correlation and NAS speedup across four benchmark families, multiple datasets, runtime budgets, and training/test distributions.The evaluation includes Pearson, Spearman, Kendall Tau, and sparse Kendall Tau correlations, alongside predictor-guided evolution and Bayesian optimization frameworks.
  • Performance Predictor Evaluation: Seven of 31 predictors are Pareto-optimal across Kendall Tau, initialization time, and query time on NAS-Bench-201 CIFAR-10.The Pareto comparison spans 154 query-time and initialization-time budgets.
  • Performance Predictor Evaluation: In low-initialization, low-query-time settings, Jacobian covariance and SynFlow perform well on NAS-Bench-101 and NAS-Bench-201, whereas zero-cost methods do not perform well on DARTS.Weight sharing also does not yield high Kendall Tau values in these search spaces.
  • Performance Predictor Evaluation: SoTL-E performs best in the low-initialization, high-query-time region, while GCN and SemiNAS lead earlier and boosted trees lead later in the high-initialization, low-query-time region.The latter region is especially competitive because many NAS algorithms query predictors rapidly after initialization.
  • Performance Predictor Evaluation: Three-second SynFlow and Jacobian covariance outperform all model-based methods even after 30 hours of initialization on NAS-Bench-101/201.The results suggest replacing model-based predictors with Jacobian covariance may improve early NAS iterations.
  • Performance Predictor Evaluation: OMNI combines SoTL-E, Jacobian covariance, and a model-based predictor, achieving strong performance across almost all budget constraints and search spaces.In some settings, OMNI reaches a Kendall Tau value 30% higher than the next-best predictors; its components capture complementary information.
  • Predictor-Based NAS Experiments: In predictor-based NAS, SemiNAS and NAO perform best for shorter runtimes, boosted trees perform best for longer runtimes, and OMNI variants often improve performance.OMNI(NGBoost) consistently outperforms NGBoost, while OMNI(SemiNAS) often achieves top performance on NAS-Bench-201 CIFAR-10 and ImageNet16-120.
  • Conclusions: The authors recommend selecting predictors by initialization and query-time budgets, then combining predictors from complementary families.For medium initialization time and low runtime on search spaces similar to NAS-Bench-201 or DARTS, they recommend NGBoost with Jacobian covariance and SynFlow features.

5 Societal Impact

The work aims to make performance-predictor development and comparison quicker and fairer for the AutoML community, while remaining one abstraction level removed from real applications.

  • Societal impact: The library can help practitioners choose performance predictors without conducting computationally intensive experiments themselves.The paper also presents the work as a resource for developing and fairly comparing new predictors.

6 Conclusions and Limitations

The paper presents a large-scale comparison of 31 NAS performance predictors and introduces OMNI, which combines complementary information from three predictor families. The authors identify broader evaluation across machine-learning tasks as an important extension.

  • 6 Conclusions and Limitations: The study compares 31 performance predictors across four search spaces and multiple metrics and settings.The evaluated families include learning-curve extrapolation, weight sharing, zero-cost, and model-based methods.
  • 6 Conclusions and Limitations: The experiments should be extended to more machine-learning tasks beyond image classification and language modeling.The current study covers four search spaces but remains within those task domains.
  • 6 Conclusions and Limitations: OMNI combines complementary information from three predictor families and achieves substantially improved performance.The authors release a predictor library to support more sophisticated combinations and follow-up research.

A NAS Research checklist

The study follows NAS research best practices by using fixed benchmarks and evaluation protocols, controlling experimental details, reporting resources and seeds, and releasing code. It also compares performance across initialization and query time, but does not compare against random search.

  • Best practices for comparing NAS methods: The experiments use NAS benchmarks with fixed training details, search spaces, hyperparameters, and evaluation protocols.These choices control several potential confounding factors across compared methods.
  • Best practices for releasing code: The authors report ablations, run 100 trials per experiment, and release code including experiment-launch details and seeds.Hyperparameters and experimental setup details are also reported in the paper and appendices.
  • Best practices for reporting important details: The study reports performance over initialization time and query time rather than only search-phase results.The paper presents results along performance, initialization-time, and query-time axes.
  • Best practices for comparing NAS methods: The study does not compare its predictors against random search.It instead uses other predictor baselines, including random forests and simple MLPs.

B Details from Section 4 (Experiments)

This section supplies additional implementation, tuning, experimental-results, and OMNI ablation details for the study’s 31 predictors.

  • B Details from Section 4 (Experiments): The appendix describes all 31 predictors, hyperparameter tuning, detailed results across search spaces, and OMNI’s pseudocode and ablation study.It also records dataset licenses for the benchmarks used.

B.1 Descriptions and Implementation Details

The appendix describes the 31 predictors used in the study, spanning model-based, zero-cost, learning-curve, and weight-sharing approaches, along with implementation choices and tuning considerations.

  • BANANAS uses an ensemble of three MLPs with path encoding for architecture representations.
  • Gaussian processes model predictions through mean and covariance functions, but their runtime is cubic in the number of datapoints.
  • Jacobian covariance measures modeling flexibility from the covariance of prediction Jacobians across image inputs.
  • Learning curve extrapolation fits partial learning curves to parametric models and selects among them using MCMC.
  • LcSVR combines partial learning curves and their derivatives with training hyperparameters, but requires both trained architectures and partial training for queries.
  • The evaluation used lightweight cross-validation and randomized hyperparameter search to reduce differences in tuning across model-based predictors.

B.3 Additional Experiments

Additional experiments show that predictor performance varies across search spaces and encodings, while some broad trends remain stable across datasets and metrics.

  • Predictor rankings are largely consistent across NAS-Bench-201 datasets but differ clearly between NAS-Bench-201, NAS-Bench-101, and DARTS.
  • BANANAS has the highest rank correlation on NAS-Bench-101 at high initialization time but the worst rank correlation on DARTS.
  • The appendix compares Pareto-optimal predictors using Pearson, Spearman, sparse Kendall Tau, and Kendall Tau across initialization and query-time budgets.
  • Zero-cost predictors could not be implemented on NAS-Bench-NLP because its code uses an older PyTorch version, and SOTL variants underperform Early Stop accuracy there.
  • Using path encoding makes simpler tree-based and GP-based predictors surpass BANANAS on NAS-Bench-101.

B.4 OMNI Details and Ablation

OMNI combines learning-curve, zero-cost, and model-based predictors, and its ablation shows that combining all three families gives the strongest and most consistent gains across budget constraints.

  • OMNI combines SoTL-E, Jacobian covariance, and NGBoost or SemiNAS from learning-curve, zero-cost, and model-based families.
  • The full NGBoost + Jacobian covariance + SoTL-E variant is consistent across budgets and peaks at a 30% Kendall Tau improvement.
  • NGBoost + Jacobian covariance and SoTL-E + NGBoost each reach 20% Kendall Tau improvements for some budget constraints but perform poorly for others.
  • The SemiNAS version of OMNI performs worse, and OMNI performs comparatively worse in the mutation-based setting than in the standard uniformly random setting.
  • OMNI trains an NGBoost model on architecture encodings, SoTL-E, and Jacobian covariance to predict final validation accuracy.
  • Among hybrid learning-curve and model-based predictors, LC-prev-builds outperforms LcSVR at small initialization times and high query times, while LCNet never outperforms LcSVR.
  • On NAS-Bench-201 CIFAR-10, flops and params tie for second-highest Kendall Tau among six zero-cost predictors, behind Jacobian covariance.

C.3 Random seed experiments

The random-seed experiments separate variability from changing train/test samples and show that predictor stochasticity differs substantially across methods, while longer tuning budgets rarely help consistently.

  • The main plots measure standard deviation across 100 trials while varying train/test sets, random seeds, and predictor stochasticity.
  • With fixed train and test sets, Bayesian Linear Regression has the highest stochasticity among eight evaluated predictors.
  • Increasing model-based hyperparameter tuning from 15 minutes to 1 hour produced no predictor with non-negligible improvement across all initialization-time settings.
  • BONAS had the lowest worst-two average at -0.020, possibly because the longer tuning budget introduced additional overfitting.

D Reproducibility Table

The reproducibility analysis checks whether published predictor results can be fairly replicated under the study’s search-space, timing, and metric settings. Of 31 predictors, 15 met these comparability criteria, and their reproduced rank correlations closely matched reported values.

  • Reproducibility procedure: The study used original implementations when possible and applied light hyperparameter tuning across methods.These steps were intended to control for differences in implementation quality and prior hyperparameter optimization.
  • Reproducibility procedure: Reproduction was assessed by matching at least one search space, initialization and query-time setting, and evaluation metric from the study.Predictors were checked against all three criteria before being considered comparable with published results.
  • Reproducibility results: 15 of the 31 predictors could be fairly compared with an original paper or the first NAS-Benchmark evaluation.Fourteen predictors predated NAS-Bench search spaces, while two others lacked experiments in the study’s setting.
  • Reproducibility results: All 15 comparable predictors were within 0.04 of their reported rank-correlation value or higher.The reproduced results were close to the originals or sometimes stronger because of hyperparameter optimization.
  • Reproducibility caveat: Some reported values were approximated from a plot, limiting precision for those comparisons.This qualification is recorded alongside the reproducibility results.
Loading 2104.01177v2…