PaperlayerView exact revision on arXiv

Source-linked AI summary

Best Practices for Scientific Research on Neural Architecture Search

Marius Lindauer, Frank Hutter

arXiv:1909.02453v3cs.LGstat.ML

TL;DR

NAS research has advanced methodologically, but its empirical evaluations remain less rigorous than those in other machine-learning areas. This paper organizes known evaluation and reproducibility issues into best practices and a checklist, while acknowledging practical limits such as computational resources and hyperparameter sensitivity. It aims to increase scientific rigor and support sustained progress in NAS.

  • Problem

    NAS empirical evaluations lag behind those in other machine-learning areas, with reproducibility and comparability concerns undermining confidence in reported results.

    Source: §1 · pp. 1–2, Best Practice 6: Use the Same… · p. 6

  • Method

    The paper proposes a prescriptive set of NAS best practices covering reproducibility, evaluation protocols, repeated runs, benchmarks, and hyperparameter reporting.

    Source: §1 · p. 2, §2 · p. 2, Best Practice 6: Use the Same… · p. 6

  • Results

    The paper produces 14 best practices and an accompanying checklist for NAS authors and reviewers.

    Source: §6 · p. 13

  • Takeaways & Limitations

    Following these guidelines is intended to increase the scientific rigor of NAS papers and help the community make sustained progress.

    Source: §6 · p. 13

    Some recommended practices, especially ablation studies and repeated runs, may be impossible on standard NAS benchmarks for researchers with limited computational resources.

    Source: Best Practice 10: Use Tabular or… · p. 8

Abstract

from arXiv · show

Finding a well-performing architecture is often tedious for both DL practitioners and researchers, leading to tremendous interest in the automation of this task by means of neural architecture search (NAS). Although the community has made major strides in developing better NAS methods, the quality of scientific empirical evaluations in the young field of NAS is still lacking behind that of other areas of machine learning. To address this issue, we describe a set of possible issues and ways to avoid them, leading to the NAS best practices checklist available at http://automl.org/nas_checklist.pdf.

1. Introduction

NAS automates the search for well-performing neural-network architectures, but its empirical evaluations lag behind standards in other machine-learning fields. The paper proposes best practices to improve reproducibility, reduce confirmation bias, and support sustained progress.

  • NAS searches for well-performing neural-network architectures for given datasets and has attracted rapidly increasing research interest.Source: §1 · p. 1
  • Empirical evaluation quality in NAS lags behind that of other machine-learning, AI, and optimization fields.Source: §1 · p. 1
  • Poor practices can make results irreproducible and increase confirmation bias, slowing overall research progress.Source: §1 · p. 2
  • The paper proposes best practices for empirical NAS evaluations to facilitate sustained and measurable progress.Source: §1 · p. 2
  • The authors avoid singling out flawed papers because no paper, including their own, is perfect.Source: §1 · p. 2

2. Best Practices for Releasing Code

Reproducible NAS research requires sharing the complete computational environment, including code and training pipelines. These materials enable others to reproduce results, understand existing methods, and compare new ideas fairly.

  • Reproducibility depends on sharing the software environment and instructions that generated the reported figures.Source: §2 · p. 2
  • Code availability helps researchers study existing methods and properly evaluate new ideas.Source: §2 · p. 2
  • Reproducing NAS experiments is often nearly impossible without code because important settings are hidden in both training pipelines and NAS methods.Source: §2 · p. 2
  • Different training pipelines can make performance results incomparable, so releasing the pipeline enables meaningful comparisons.Source: Best Practice 1: Release Code for the… · p. 3
  • Releasing NAS-method code additionally allows others to apply methods to new datasets and build upon the codebase.Source: Best Practice 2: Release Code for Your… · p. 3

Best Practice 3: Don’t Wait Until You’ve Cleaned up the Code; That Time May Never Come

The paper encourages researchers to release research code as used rather than delaying publication until it is cleaned up. Fair comparisons also require consistent benchmarks and transparent experimental components.

Best Practice 3: Don’t Wait Until You’ve Cleaned up the Code; That Time May Never Come

Best Practice 4: Use the Same NAS Benchmarks, not Just the Same Datasets

Best Practice 5: Run Ablation Studies

NAS methods contain many interacting components, so performance improvements require ablation studies that isolate the importance of individual benchmark and method changes.

Best Practice 6: Use the Same Evaluation Protocol for the Methods Being Compared

NAS evaluations should use the same clearly defined compute and performance protocol when comparing methods. The protocol must state whether it measures architecture identification or the full AutoML process and should account for compute over time.

Evaluation protocol

  • Different NAS evaluation schemes can make results incomparable, especially when methods differ in how many architectures they sample and evaluate.The paper contrasts returning a single final architecture with sampling thousands of architectures before selecting one.Source: Best Practice 6: Use the Same… · p. 6
  • Performance should be reported as a function of compute resources by tracking the incumbent architecture at each stopping time.Most NAS methods are anytime algorithms, so evaluations can report the architecture that would be returned if the search stopped at time t.Source: Best Practice 7: Evaluate Performance… · p. 6

Evaluation variants

  • The architecture identification variant counts only the resources needed to identify and return the final architecture.Offline evaluations plot the current incumbent architecture after applying the final evaluation pipeline.Source: Best Practice 7: Evaluate Performance… · pp. 6–7
  • The AutoML variant also counts training the final architecture and evaluates the incumbent model without retraining at each compute step.This reflects a setting where the system must be ready to make predictions when the compute budget ends.Source: Best Practice 7: Evaluate Performance… · p. 7
  • The AutoML variant is more suitable when the objective is a good ready-to-use model for a dataset, although architecture identification remains more common.The architecture identification variant is widely used because NAS has seen limited use in full AutoML settings.Source: Best Practice 7: Evaluate Performance… · p. 7

Best Practice 8: Compare Against Random Sampling and Random Search

NAS papers should compare against both random sampling and random search because these simple baselines can be strong and distinguish search-method benefits from search-space quality.

Baseline definitions

  • Random sampling and random search are distinct baselines despite both drawing uniform random samples from the architecture space.Random sampling returns one sample, whereas random search evaluates samples and tracks the best incumbent.Source: Best Practice 8: Compare Against… · p. 7

Random sampling

  • Random sampling has essentially zero runtime under the architecture-identification definition and an expected performance equal to the search space’s average architecture performance.It draws and returns a single architecture without iterative evaluation.Source: Best Practice 8: Compare Against… · p. 7

Random search

  • Random search is an anytime procedure that should run for the same duration and use the same evaluation criterion as the compared NAS method.It repeatedly evaluates random architectures and returns the best incumbent when stopped.Source: Best Practice 8: Compare Against… · p. 7

Why compare

  • Random sampling can perform strongly in well-designed search spaces, while random search can be highly competitive.The paper therefore recommends comparing against both baselines.Source: Best Practice 8: Compare Against… · p. 7

Best Practice 9: Perform Multiple Runs with Different Seeds

NAS methods are stochastic, so reliable evaluation requires repeated runs with different seeds and careful control of confounding factors. When compute limits prevent this, tabular or surrogate benchmarks can support more extensive evaluation, with exceptions for some methods.

Repeated runs

  • Re-running a NAS method on the same dataset can produce different results because NAS methods are usually stochastic.Even available source code may require several runs to reproduce published results.Source: Best Practice 9: Perform Multiple Runs… · p. 8

Compute constraints

  • Tabular and surrogate NAS benchmarks enable extensive evaluations when computational limits make repeated runs and ablation studies on standard benchmarks impractical.The paper advocates using these benchmarks especially under limited computational resources.Source: Best Practice 10: Use Tabular or… · p. 8

Method-dependent limits

  • Weight-sharing and weight-inheritance methods still require substantial search computation on tabular or surrogate benchmarks, so many runs should not be expected for them.Black-box and multi-fidelity methods can use efficient lookups, but these methods cannot similarly speed up the search phase.Source: Best Practice 10: Use Tabular or… · p. 9

Confounding factors

  • Different hardware, runtimes, and deep-learning library versions can make results incomparable even on the same NAS benchmark.The paper recommends controlling these confounding factors as much as possible.Source: Best Practice 11: Control Confounding… · p. 9

Best Practice 12: Report the Use of Hyperparameter Optimization

NAS studies should report hyperparameter optimization because tuning can become part of the method and, in AutoML settings, part of its runtime. Compute and implementation details should also be reported end to end for reproducible comparisons.

Hyperparameter optimization

  • A NAS method’s hyperparameter optimization strategy should be reported rather than treated as an incidental tuning step.The paper warns that replacing manual architecture engineering with manual NAS-method hyperparameter optimization raises a methodological concern.Source: Best Practice 12: Report the Use of… · pp. 9–10
  • In an AutoML evaluation, hyperparameter optimization is part of the NAS method and should count toward its runtime.AutoML is expected to be robust, making the optimization strategy part of the system being evaluated.Source: Best Practice 12: Report the Use of… · p. 10

End-to-end accounting

  • Compute requirements should be measured end to end across parallel searches, validation, and final training.For k searches, the architecture-identification total is k · (Tsearch + Tvalid), while the AutoML total adds Tfinal.Source: Best Practice 13: Report End-to-End… · p. 10

Reproducibility details

  • Reported speed comparisons require the hardware and deep-learning library versions used for evaluation.Different GPUs and library versions can substantially affect runtime and reproducibility.Source: Best Practice 14: Report All the… · p. 10
  • Publishing a container such as Docker or Singularity can help preserve publicly available software dependencies.This is presented as one way to avoid omitting dependencies when releasing code.Source: Best Practice 14: Report All the… · p. 10

The Need for Proper NAS Benchmarks

NAS research relies heavily on a small set of datasets and search spaces, creating risks of overfitting and limiting generalization. The paper therefore advocates a diverse, standardized, reproducible benchmark ecosystem spanning applications, search spaces, training pipelines, and benchmark types.

  • CIFAR-10 and PTB dominate NAS evaluations, risking overfitting because they function as a two-sample meta-learning training set.Source: The Need for Proper NAS Benchmarks · p. 11
  • A larger standardized suite of NAS benchmarks would enable comparisons under nearly identical conditions and help control confounding factors.The paper defines a benchmark as including a dataset, search space, training pipeline, source code, and known hyperparameters.Source: The Need for Proper NAS Benchmarks · p. 11
  • Application-focused NAS papers should create fully reproducible benchmarks, which the authors regard as potentially more impactful than slight NAS-method improvements.Source: The Need for Proper NAS Benchmarks · p. 11

Properties of a NAS Benchmark

  • NAS benchmark libraries should vary in difficulty, expressive power, complexity, novelty, applications, datasets, and training pipelines.The proposed diversity also includes real, tabular, and surrogate benchmarks.Source: Properties of a NAS Benchmark · p. 12
  • Tabular benchmarks should be easy to access, accompanied by generation code and hyperparameters, and retain additional metrics such as weights and FLOPs.Surrogate benchmarks additionally require careful data collection, validation, versioning, and release of training data and source code.Source: Practical Recommendations for Tabular… · pp. 12–13

The Need for an Open-Source Library of NAS Methods

The paper calls for an open-source library of NAS methods to make comparisons fairer and experiments easier to control. Such a library would also support assessing the importance of individual method components.

6. Conclusion

The paper proposes 14 best practices and compiles them into a checklist for authors and reviewers. It presents these guidelines as a means to strengthen NAS research rigor and support sustained progress.

  • The paper proposes 14 best practices for scientific research on NAS methods.Source: §6 · p. 13
  • The practices are compiled into a checklist intended to help authors and reviewers assess the state of NAS papers.The checklist is available at http://automl.org/NAS_checklist.pdf.Source: §6 · p. 13

Related papers

Loading 1909.02453v3…