Source-linked AI summary
Best Practices for Scientific Research on Neural Architecture Search
Marius Lindauer, Frank Hutter
TL;DR
NAS research has advanced methodologically, but its empirical evaluations remain less rigorous than those in other machine-learning areas. This paper organizes known evaluation and reproducibility issues into best practices and a checklist, while acknowledging practical limits such as computational resources and hyperparameter sensitivity. It aims to increase scientific rigor and support sustained progress in NAS.
Problem
NAS empirical evaluations lag behind those in other machine-learning areas, with reproducibility and comparability concerns undermining confidence in reported results.
Method
The paper proposes a prescriptive set of NAS best practices covering reproducibility, evaluation protocols, repeated runs, benchmarks, and hyperparameter reporting.
Results
The paper produces 14 best practices and an accompanying checklist for NAS authors and reviewers.
Takeaways & Limitations
Following these guidelines is intended to increase the scientific rigor of NAS papers and help the community make sustained progress.
Takeaways & Limitations
Some recommended practices, especially ablation studies and repeated runs, may be impossible on standard NAS benchmarks for researchers with limited computational resources.
Abstract
from arXiv · showhide
Finding a well-performing architecture is often tedious for both DL practitioners and researchers, leading to tremendous interest in the automation of this task by means of neural architecture search (NAS). Although the community has made major strides in developing better NAS methods, the quality of scientific empirical evaluations in the young field of NAS is still lacking behind that of other areas of machine learning. To address this issue, we describe a set of possible issues and ways to avoid them, leading to the NAS best practices checklist available at http://automl.org/nas_checklist.pdf.
1. Introduction
NAS automates the search for well-performing neural-network architectures, but its empirical evaluations lag behind standards in other machine-learning fields. The paper proposes best practices to improve reproducibility, reduce confirmation bias, and support sustained progress.
- NAS searches for well-performing neural-network architectures for given datasets and has attracted rapidly increasing research interest.
- Empirical evaluation quality in NAS lags behind that of other machine-learning, AI, and optimization fields.
- Poor practices can make results irreproducible and increase confirmation bias, slowing overall research progress.
- The paper proposes best practices for empirical NAS evaluations to facilitate sustained and measurable progress.
- The authors avoid singling out flawed papers because no paper, including their own, is perfect.
2. Best Practices for Releasing Code
Reproducible NAS research requires sharing the complete computational environment, including code and training pipelines. These materials enable others to reproduce results, understand existing methods, and compare new ideas fairly.
- Reproducibility depends on sharing the software environment and instructions that generated the reported figures.
- Code availability helps researchers study existing methods and properly evaluate new ideas.
- Reproducing NAS experiments is often nearly impossible without code because important settings are hidden in both training pipelines and NAS methods.
- Different training pipelines can make performance results incomparable, so releasing the pipeline enables meaningful comparisons.
- Releasing NAS-method code additionally allows others to apply methods to new datasets and build upon the codebase.
Best Practice 3: Don’t Wait Until You’ve Cleaned up the Code; That Time May Never Come
The paper encourages researchers to release research code as used rather than delaying publication until it is cleaned up. Fair comparisons also require consistent benchmarks and transparent experimental components.
- Best Practice 3: Don’t Wait Until You’ve Cleaned up the Code; That Time May Never Come: Researchers should publish prototype research code promptly instead of waiting for a later cleanup that may never happen.
- Best Practice 3: Don’t Wait Until You’ve Cleaned up the Code; That Time May Never Come: Reproducibility has become increasingly prominent, with major venues requesting authors to complete reproducibility checklists.
- Best Practice 3: Don’t Wait Until You’ve Cleaned up the Code; That Time May Never Come: Sharing code may provide research labs a competitive advantage in publishing and hiring, although proprietary dependencies can make release difficult.
- Best Practice 4: Use the Same NAS Benchmarks, not Just the Same Datasets: Reported NAS results are often incomparable across different search spaces, training procedures, and regularization techniques, even on the same dataset.
- Best Practice 4: Use the Same NAS Benchmarks, not Just the Same Datasets: A NAS benchmark combines a dataset split, search space, runnable code, and predefined training hyperparameters.
- Best Practice 4: Use the Same NAS Benchmarks, not Just the Same Datasets: Tabular benchmarks replace repeated neural-network training with table lookup, while surrogate benchmarks use performance-prediction models for larger spaces.
Best Practice 5: Run Ablation Studies
NAS methods contain many interacting components, so performance improvements require ablation studies that isolate the importance of individual benchmark and method changes.
- NAS papers often modify benchmarks, pipelines, or method components without clarifying which change produced the final performance.
- The field still lacks thorough insight into which aspects of NAS matter most.
- Ablation analyses should study individual components affecting performance, including changes to both NAS benchmarks and NAS methods.
Best Practice 6: Use the Same Evaluation Protocol for the Methods Being Compared
NAS evaluations should use the same clearly defined compute and performance protocol when comparing methods. The protocol must state whether it measures architecture identification or the full AutoML process and should account for compute over time.
- Evaluation protocol: Different NAS evaluation schemes can make results incomparable, especially when methods differ in how many architectures they sample and evaluate.The paper contrasts returning a single final architecture with sampling thousands of architectures before selecting one.
- Evaluation protocol: Performance should be reported as a function of compute resources by tracking the incumbent architecture at each stopping time.Most NAS methods are anytime algorithms, so evaluations can report the architecture that would be returned if the search stopped at time t.
- Evaluation variants: The architecture identification variant counts only the resources needed to identify and return the final architecture.Offline evaluations plot the current incumbent architecture after applying the final evaluation pipeline.
- Evaluation variants: The AutoML variant also counts training the final architecture and evaluates the incumbent model without retraining at each compute step.This reflects a setting where the system must be ready to make predictions when the compute budget ends.
- Evaluation variants: The AutoML variant is more suitable when the objective is a good ready-to-use model for a dataset, although architecture identification remains more common.The architecture identification variant is widely used because NAS has seen limited use in full AutoML settings.
Best Practice 8: Compare Against Random Sampling and Random Search
NAS papers should compare against both random sampling and random search because these simple baselines can be strong and distinguish search-method benefits from search-space quality.
- Baseline definitions: Random sampling and random search are distinct baselines despite both drawing uniform random samples from the architecture space.Random sampling returns one sample, whereas random search evaluates samples and tracks the best incumbent.
- Random sampling: Random sampling has essentially zero runtime under the architecture-identification definition and an expected performance equal to the search space’s average architecture performance.It draws and returns a single architecture without iterative evaluation.
- Random search: Random search is an anytime procedure that should run for the same duration and use the same evaluation criterion as the compared NAS method.It repeatedly evaluates random architectures and returns the best incumbent when stopped.
- Why compare: Random sampling can perform strongly in well-designed search spaces, while random search can be highly competitive.The paper therefore recommends comparing against both baselines.
Best Practice 9: Perform Multiple Runs with Different Seeds
NAS methods are stochastic, so reliable evaluation requires repeated runs with different seeds and careful control of confounding factors. When compute limits prevent this, tabular or surrogate benchmarks can support more extensive evaluation, with exceptions for some methods.
- Repeated runs: Re-running a NAS method on the same dataset can produce different results because NAS methods are usually stochastic.Even available source code may require several runs to reproduce published results.
- Compute constraints: Tabular and surrogate NAS benchmarks enable extensive evaluations when computational limits make repeated runs and ablation studies on standard benchmarks impractical.The paper advocates using these benchmarks especially under limited computational resources.
- Method-dependent limits: Weight-sharing and weight-inheritance methods still require substantial search computation on tabular or surrogate benchmarks, so many runs should not be expected for them.Black-box and multi-fidelity methods can use efficient lookups, but these methods cannot similarly speed up the search phase.
- Confounding factors: Different hardware, runtimes, and deep-learning library versions can make results incomparable even on the same NAS benchmark.The paper recommends controlling these confounding factors as much as possible.
Best Practice 12: Report the Use of Hyperparameter Optimization
NAS studies should report hyperparameter optimization because tuning can become part of the method and, in AutoML settings, part of its runtime. Compute and implementation details should also be reported end to end for reproducible comparisons.
- Hyperparameter optimization: A NAS method’s hyperparameter optimization strategy should be reported rather than treated as an incidental tuning step.The paper warns that replacing manual architecture engineering with manual NAS-method hyperparameter optimization raises a methodological concern.
- Hyperparameter optimization: In an AutoML evaluation, hyperparameter optimization is part of the NAS method and should count toward its runtime.AutoML is expected to be robust, making the optimization strategy part of the system being evaluated.
- End-to-end accounting: Compute requirements should be measured end to end across parallel searches, validation, and final training.For k searches, the architecture-identification total is k · (Tsearch + Tvalid), while the AutoML total adds Tfinal.
- Reproducibility details: Reported speed comparisons require the hardware and deep-learning library versions used for evaluation.Different GPUs and library versions can substantially affect runtime and reproducibility.
- Reproducibility details: Publishing a container such as Docker or Singularity can help preserve publicly available software dependencies.This is presented as one way to avoid omitting dependencies when releasing code.
The Need for Proper NAS Benchmarks
NAS research relies heavily on a small set of datasets and search spaces, creating risks of overfitting and limiting generalization. The paper therefore advocates a diverse, standardized, reproducible benchmark ecosystem spanning applications, search spaces, training pipelines, and benchmark types.
- CIFAR-10 and PTB dominate NAS evaluations, risking overfitting because they function as a two-sample meta-learning training set.
- A larger standardized suite of NAS benchmarks would enable comparisons under nearly identical conditions and help control confounding factors.The paper defines a benchmark as including a dataset, search space, training pipeline, source code, and known hyperparameters.
- Application-focused NAS papers should create fully reproducible benchmarks, which the authors regard as potentially more impactful than slight NAS-method improvements.
- Properties of a NAS Benchmark: NAS benchmark libraries should vary in difficulty, expressive power, complexity, novelty, applications, datasets, and training pipelines.The proposed diversity also includes real, tabular, and surrogate benchmarks.
- Tabular benchmarks should be easy to access, accompanied by generation code and hyperparameters, and retain additional metrics such as weights and FLOPs.Surrogate benchmarks additionally require careful data collection, validation, versioning, and release of training data and source code.
The Need for an Open-Source Library of NAS Methods
The paper calls for an open-source library of NAS methods to make comparisons fairer and experiments easier to control. Such a library would also support assessing the importance of individual method components.
- An open-source NAS-method library should provide a common interface, control confounding factors, enable fair comparisons across benchmarks, and support component assessment.
6. Conclusion
The paper proposes 14 best practices and compiles them into a checklist for authors and reviewers. It presents these guidelines as a means to strengthen NAS research rigor and support sustained progress.
- The paper proposes 14 best practices for scientific research on NAS methods.
- The practices are compiled into a checklist intended to help authors and reviewers assess the state of NAS papers.The checklist is available at http://automl.org/NAS_checklist.pdf.