PaperlayerView exact revision on arXiv

Source-linked AI summary

Random Search and Reproducibility for Neural Architecture Search

Liam Li, Ameet Talwalkar

arXiv:1902.07638v3cs.LGstat.ML

TL;DR

NAS lacks adequate grounding from simple hyperparameter-optimization baselines and reproducible evaluation. The paper tests random search with early-stopping and weight-sharing on PTB and CIFAR-10, finding competitive or state-of-the-art results while documenting reproducibility concerns and providing materials for exact replication.

  • Problem

    NAS research needs stronger baseline comparisons and more reproducible evidence because existing methods are complex and published results may lack sufficient source materials.

  • Method

    The paper evaluates random search with early-stopping and weight-sharing on PTB and CIFAR-10, using shared search spaces and reporting repeated trials and reproducibility materials.

  • Results

    Random search with early-stopping performs at least as well as ENAS on both benchmarks, while weight-sharing reaches state-of-the-art PTB performance and is highly competitive on CIFAR-10.

  • Takeaways & Limitations

    Random search provides a competitive NAS baseline, and weight-sharing can match or challenge more complicated NAS methods under similar computational budgets.

  • Takeaways & Limitations

    Weight-sharing performance estimates can be noisy, and evaluating more architectures increases computational time.

Abstract

from arXiv · show

Neural architecture search (NAS) is a promising research direction that has the potential to replace expert-designed networks with learned, task-specific architectures. In this work, in order to help ground the empirical results in this field, we propose new NAS baselines that build off the following observations: (i) NAS is a specialized hyperparameter optimization problem; and (ii) random search is a competitive baseline for hyperparameter optimization. Leveraging these observations, we evaluate both random search with early-stopping and a novel random search with weight-sharing algorithm on two standard NAS benchmarks---PTB and CIFAR-10. Our results show that random search with early-stopping is a competitive NAS baseline, e.g., it performs at least as well as ENAS, a leading NAS method, on both benchmarks. Additionally, random search with weight-sharing outperforms random search with early-stopping, achieving a state-of-the-art NAS result on PTB and a highly competitive result on CIFAR-10. Finally, we explore the existing reproducibility issues of published NAS results. We note the lack of source material needed to exactly reproduce these results, and further discuss the robustness of published results given the various sources of variability in NAS experimental setups. Relatedly, we provide all information (code, random seeds, documentation) needed to exactly reproduce our results, and report our random search with weight-sharing results for each benchmark on multiple runs.

1 INTRODUCTION

The paper frames NAS as specialized hyperparameter optimization and argues that stronger, simpler baselines and reproducible reporting are needed. It evaluates random search variants and identifies substantial reproducibility concerns in existing NAS results.

  • NAS aims to replace painstaking expert-designed architectures with automatically identified architectures that can outperform hand-designed ones.
  • Inadequate Baselines: Existing NAS comparisons inadequately evaluate standard hyperparameter optimization methods despite NAS being a specialized hyperparameter optimization problem.
  • Complex Methods: NAS methods often introduce complex training routines, architecture transformations, and modeling assumptions without ablations isolating which components are necessary.
  • Lack of Reproducibility: NAS reproducibility includes exact replication of reported results and broad robustness across experimental variation, but published work often lacks needed materials.
  • The study provides code, random seeds, and documentation, repeats weight-sharing experiments across seeds, and observes non-trivial differences across runs.
  • 56.4 test perplexity on PTB and 2.85% test error on CIFAR-10 make random search with early-stopping competitive with ENAS using comparable compute.
  • Random search with weight-sharing achieves state-of-the-art PTB performance and comparable CIFAR-10 performance to DARTS using similar computational budgets.

2 RELATED WORK

The section identifies complexity and weak comparison practices in NAS, then motivates random search and ASHA as simpler, applicable baselines. It also examines reproducibility, finding that exact reproduction requires four criteria and that published NAS results often lack sufficient materials or repeated-run evidence.

  • Hyperparameter optimization baselines: Existing NAS studies provide inadequate comparisons with random search and other hyperparameter-optimization methods.Some compare against random search using only a few evaluations, while others use Bayesian optimization without efficient architecture evaluation schemes.
  • Hyperparameter optimization baselines: ASHA combines random search with early stopping and is selected as a competitive baseline because it applies readily to NAS problems.The passage characterizes ASHA as a state-of-the-art, theoretically principled, bandit-based method.
  • NAS search methods: NAS methods add substantial search complexity through specialized mutations, kernels, transformations, and gradient procedures, without a clear overall winner.These adaptations also complicate comparisons because methods may use different search spaces and evaluation methods.
  • NAS search methods: Random search samples architectures directly from the search space to simplify NAS and isolate important components.The authors use random search as a deliberately simple search process.
  • NAS evaluation methods: Weight-sharing methods use one set of weights across architectures but require careful training routines, adding complexity to NAS evaluation.Other evaluation approaches likewise introduce architecture transformations or auxiliary networks that encode information from previously seen architectures.
  • Reproducibility: Exact reproducibility requires architecture-search code, model-evaluation code, hyperparameter-tuning documentation, and random seeds, yet none of the listed methods meet all four criteria.Random seeds are needed for search and post-processing, and are useful only when the associated code is deterministic up to the seed.
  • Reproducibility: Published NAS work commonly reports only the best architecture from a single search run, limiting evidence about broad reproducibility amid high variance in extremal results.The section notes minimal discussion of broad reproducibility, with NASBOT and DARTS as exceptions among the listed methods.

3 METHODOLOGY

The paper combines random search with weight-sharing for DAG-based NAS, training shared weights with sampled architectures and selecting the best architecture using those weights. It also analyzes meta-hyperparameters governing search quality, cost, memory, and exploration.

  • Random Search with Weight-Sharing: The algorithm trains shared weights by selecting one randomly sampled architecture per minibatch and back-propagating only through its activated edges and operations.The number of architectures used to update shared weights equals the total number of minibatch training iterations.
  • Random Search with Weight-Sharing: After shared-weight training, the method evaluates randomly sampled architectures on held-out data and selects the best-performing architecture.
  • Meta-Hyperparameters: Increasing training epochs or decreasing batch size increases minibatch updates and architecture exposure, but also increases search cost; smaller batches produce noisier gradients.
  • Meta-Hyperparameters: Larger search networks can store more information about architectures but require more GPU memory, while evaluating more architectures increases exploration and computational cost.
  • Memory Footprint: The method can load only activated operations and edges, reducing its memory footprint to that of a single model and enabling weight-sharing search on larger proxyless networks.The paper contrasts this with path dropout, where many edges remain active early in training.

4 EXPERIMENTS

Experiments evaluate the proposed methods on PTB language modeling and CIFAR-10 image classification using DARTS-compatible search spaces and staged evaluation procedures. Random search with early-stopping uses partial training to select architectures before final evaluation.

  • Benchmarks: The experiments use PTB and CIFAR-10, retaining the same search spaces and much of DARTS’s experimental setup to facilitate comparison.
  • Weight-Sharing Evaluation: Weight-sharing evaluation has three stages: search a cell on a cheaper task, retrain the best architecture in a larger network, then conduct full evaluation with more epochs or seeds.
  • Early-Stopping Evaluation: For ASHA, architectures receive partial training beginning at r = 1 epoch, with a maximum of 300 epochs and promotion rate η = 4.At each round, the top 1/4 of architectures are promoted and trained for 4× more resource.

4.1 PTB Benchmark

On PTB, ASHA is competitive with leading NAS baselines at comparable cost, while tuned random search with weight-sharing reaches a stronger result. Reproducibility analyses reveal sensitivity to seeds, convergence rates, and intermediate rankings.

  • Final Search Results: 56.4 test perplexity: ASHA evaluated 300 architectures using 2 GPU days and performed comparably to published ENAS while outperforming several random-search and DARTS baselines.The 2 GPU-day budget matched the total cost of second-order DARTS.
  • Final Search Results: 55.5 test perplexity: tuned random search with weight-sharing achieved state-of-the-art perplexity with slightly lower search cost than DARTS.The comparison excludes mixture-of-experts softmax training to remain fair to ENAS and DARTS.
  • Impact of Meta-Hyperparameters: Random search with weight-sharing tunes gradient clipping, batch size, and network size, and selects the best architecture from evaluations of 2000 randomly sampled architectures.
  • Impact of Meta-Hyperparameters: Stage (2) results compare settings across four trials after evaluating 2000 sampled architectures and retraining each trial’s best architecture from scratch for 300 epochs.
  • Investigating Reproducibility: Intermediate architecture rankings can swap across training durations: 300 epochs and shared-weight evaluation may fail to recover the ranking found after full training.Different convergence rates explain why architectures with comparable intermediate perplexity can have different final test perplexities.
  • Investigating Reproducibility: Independent trials show high variance in intermediate results, and stage (3) used DARTS-tuned training hyperparameters without additional tuning for the selected architecture.

4.2 CIFAR-10 Benchmark

On CIFAR-10, the study compares random-search baselines and weight-sharing methods under DARTS-like evaluation, then examines their reproducibility and tuning sensitivity. Random search with early-stopping and weight-sharing is competitive with leading NAS methods, but performance varies across runs and intermediate rankings can be unstable.

  • Search setup: The CIFAR-10 search space uses convolutional cells with four search nodes and candidate separable convolutions, dilated convolutions, pooling, and zero operations.Each node selects two prior inputs and associated operations.
  • Search setup: A proxy network with 8 stacked cells and 16 initial channels is used for search, while a proxyless network with 20 cells and 36 channels is used for evaluation.The smaller proxy network addresses the higher memory requirements of weight-sharing.
  • Final search results: 3.03 ± 0.13 average error is achieved by ASHA with 9 GPU days, comparable to DARTS and better than the cited random-search baseline.The best ASHA seed reaches 2.85 test error, lower than the published ENAS result.
  • Final search results: 2.85 ± 0.08 average test error is achieved by tuned random search with weight-sharing, comparable to SNAS, DARTS, and GHN.The manually tuned architectures outperform it but use more than 7× as many parameters.
  • Reproducibility: DARTS exact reproducibility was infeasible because its code was nondeterministic and the published search process lacked random seeds.An independent run reached 2.78 ± 0.12 versus the published 2.76 ± 0.09.
  • Reproducibility: Random search with weight-sharing achieved below 2.90 test error in 2 of 5 additional runs and averaged 2.92 across all 6 runs.The authors verified exact reproducibility before assessing broad reproducibility across independent runs.
  • Reproducibility: DARTS architecture rankings were unstable between 100 and 600 training epochs, motivating 600-epoch training for final selection.The authors caution against relying on noisy short-training signals during architecture search.
  • Reproducibility: Random search with weight-sharing showed high variance between runs, although the three underperforming additional runs still beat ASHA.Two of five additional runs matched the first run’s performance.

4.3 Computational Cost

The study compares computational cost with the quality of architecture-performance signals, finding orders-of-magnitude differences among full training, partial training, and weight-sharing. Weight-sharing is inexpensive per evaluated architecture, but its noisy estimates and unclear equivalent search coverage complicate cost comparisons.

  • Cost–signal trade-off: Evaluation methods differ by at least an order of magnitude in per-architecture cost, creating a trade-off between computational expense and signal quality.The comparison spans full training, partial training, and weight-sharing.
  • Evaluation costs: Random search with weight-sharing has an estimated amortized cost of 0.2 minutes per PTB architecture and 0.8 minutes per CIFAR-10 architecture.The estimate divides total search cost by the number of architectures evaluated using shared weights.
  • Comparability limits: The equivalent number of architectures evaluated by weight-sharing methods is difficult to quantify because shared-weight performance estimates can be noisy.For DARTS, the information supplied by gradient updates to architecture-mixture weights is also unclear.
  • Tuning and cost: Default ASHA settings achieved nearly competitive performance with roughly the computation used by DARTS and random search with weight-sharing, despite only light tuning.The authors contrast this with uncertainty about how much tuning DARTS settings required.

4.4 Available Code

The authors provide code, random seeds, final architectures, and documentation intended to make their experiments exactly reproducible under fixed seeds.

  • Reproducibility resources: The reported results are exactly reproducible from architecture search through final evaluation using the released code.The random-search weight-sharing implementation is deterministic when conditioned on a fixed random seed.
  • Reproducibility resources: The release includes final architectures and the random seeds used to discover them.These materials cover the trials reported in the paper’s tables.

5 CONCLUSION

The paper argues for stronger NAS baselines, component-isolating ablations, and reproducible evaluations. It finds that random search can closely match specialized NAS methods while independent runs and reporting practices remain important.

  • Better baselines: Random search with early-stopping reveals a surprisingly small performance gap between general-purpose hyperparameter optimization and specialized NAS methods.The paper proposes measuring novel NAS methods by their multiplicative speedup over random search with early-stopping.
  • Ablation studies: Evaluating early-stopping and weight-sharing variants isolates performance gains associated with the cheaper weight-sharing evaluation scheme.The comparison also motivates ablations of search spaces, search methods, and shared-weight evaluation.
  • Ablation studies: Random search with weight-sharing is comparable to leading NAS methods, questioning the necessity of auxiliary networks and complicated algorithmic components.The paper specifically names GHN, ENAS, SNAS, and DARTS as methods whose additional components merit isolation.
  • Reproducible results: NAS results should be reported over several independent runs because extremal statistics and substantial computational variability can make single-run comparisons unreliable.The paper notes that generating its own experiments required several months and tens of thousands of dollars.
  • Reproducible results: The paper calls for transparent intermediate-result reporting, robust performance statistics, greater computational resources, or more tractable benchmarks.It also recommends evaluating NAS methods for robustness across independent runs, not accuracy alone.

A APPENDIX

The appendix details how the experiments reproduce and approximate DARTS evaluation procedures. It specifies operation handling and the training schedule used for final architecture evaluation.

  • Experimental details: The experiments provide additional detail for the procedures described in Section 4.1.
  • Architecture Operations: DARTS includes the zero operation during shared-weight training but removes it before selecting the final architecture for evaluation.The described DARTS procedure separates the search-stage operation set from the final evaluation set.
  • Architecture Operations: Random search with weight-sharing excludes the zero operation during both search and evaluation.
  • Stage 3 Procedure: The final-network evaluation trains for an additional 1000 epochs, reaching 3600 total epochs to approximate convergence.This follows the ArXiv version of DARTS, which reported 1600-epoch and additional-1000-epoch results.

A.2 CIFAR-10 Benchmark

The CIFAR-10 appendix describes operation choices, a staged architecture-selection procedure, and determinism fixes. It also records that one ASHA result remains non-reproducible because of nondeterministic training and asynchronous updates.

  • Experimental details: The appendix provides additional detail for the experiments described in Section 4.2.
  • Architecture Operations: For random search with weight-sharing, the zero operation is included during both search and evaluation.
  • Stage 1 Procedure: Architectures are first evaluated with shared weights on 10 minibatches, retaining the best 10 from each set of 1000 candidates.The procedure then selects the top architecture across all sets for stage (2) evaluation.
  • Reproducibility: The released DARTS code produced nondeterministic CNN results because of CuDNN and data-loading behavior.The appendix identifies nondeterminism in both the computational backend and data-loading process.
  • Reproducibility: Determinism is improved by configuring CuDNN, setting seeds for random, NumPy, and PyTorch, and disabling multiple data-loading threads.
  • Reproducibility: The ASHA result does not satisfy exact reproducibility because its training is nondeterministic and uses asynchronous updates.Other random-search-with-weight-sharing settings are described as deterministic after seeding fixes.
Loading 1902.07638v3…