Source-linked AI summary
Neural Predictor for Neural Architecture Search
Wei Wen, Hanxiao Liu, Hai Li, Yiran Chen, Gabriel Bender, Pieter-Jan Kindermans
TL;DR
NAS methods can be computationally costly or implementation-heavy, motivating a simpler way to identify promising architectures. The Neural Predictor trains a regression model from randomly sampled architectures, screens many more candidates, and validates the top K. It strongly outperforms random search on NASBench-101, is about 22.83 times more sample-efficient than Regularized Evolution, and matches ProxylessNAS quality in its search space.
Problem
NAS methods improve accuracy but often require high computational cost or nontrivial specialized implementations.
Method
The Neural Predictor trains N random architectures for regression, predicts many random architectures, and validates the top K before deployment.
Results
The method strongly outperforms random search on NASBench-101, is about 22.83 times more sample-efficient than Regularized Evolution, and matches ProxylessNAS quality.
Takeaways & Limitations
Random sampling and supervised learning provide a simpler architecture-selection process than reinforcement learning, weight sharing, or Bayesian optimization.
Takeaways & Limitations
Method choice depends on search-space implementation complexity and available computational resources.
Abstract
from arXiv · showhide
Neural Architecture Search methods are effective but often use complex algorithms to come up with the best architecture. We propose an approach with three basic steps that is conceptually much simpler. First we train N random architectures to generate N (architecture, validation accuracy) pairs and use them to train a regression model that predicts accuracy based on the architecture. Next, we use this regression model to predict the validation accuracies of a large number of random architectures. Finally, we train the top-K predicted architectures and deploy the model with the best validation result. While this approach seems simple, it is more than 20 times as sample efficient as Regularized Evolution on the NASBench-101 benchmark and can compete on ImageNet with more complex approaches based on weight sharing, such as ProxylessNAS.
1. Introduction
The Neural Predictor addresses the computational cost and implementation complexity of NAS by using random sampling and supervised regression to curate promising architectures before final validation. It outperforms random search on NASBench-101 and is substantially more sample-efficient than Regularized Evolution, while matching ProxylessNAS quality in its search space.
- NAS methods improve accuracy but can require high computational cost and specialized implementations, especially when using weight sharing.
- Exhaustively training 400,000 architectures in the NASBench search space would require roughly 25 years of TPU training time.
- The Neural Predictor trains N random architectures, fits a regression model, predicts many random architectures, and finally validates the top K.
- Predictor-based evaluation is cheap, while final validation trains only a curated set of K models before deployment.
- The Neural Predictor strongly outperforms random search on NASBench-101 and is about 22.83 times more sample-efficient than Regularized Evolution.
- In the ProxylessNAS search space, the predicted architecture is as accurate as ProxylessNAS and clearly better than random search.
- The architecture-selection process uses random sampling and supervised learning rather than reinforcement learning, weight sharing, or Bayesian optimization.
2. Neural Predictor
The Neural Predictor learns architecture-to-accuracy mappings, screens many random architectures cheaply, and validates only the most promising candidates. Its graph-based predictor represents directed architectures with bidirectional information flow, while compute allocation and hyperparameters remain practical design considerations.
- The Neural Predictor uses actual training and validation for reliable selection while curating a smaller list of promising models for final validation.
- Neural Predictor workflow: Step 1 trains N models to create architecture–validation accuracy pairs and fits a regression model mapping architectures to predicted accuracy.
- Neural Predictor workflow: Step 2 rapidly predicts many random architectures and selects the top K for final validation because predictor evaluation is efficient and trivially parallelizable.
- Neural Predictor workflow: Step 3 trains and validates the K selected models, then chooses the model with the highest actual validation accuracy for deployment.
- Hyper-parameters in the Workflow: For a fixed compute budget, increasing N can improve predictor accuracy but requires decreasing K.
- Modeling by Graph Convolutional Networks: The predictor uses graph convolutional networks with one-hot operation representations, adjacency-based updates, normalization, self-cycles, and forward and reverse information flow.
- Modeling by Graph Convolutional Networks: Averaging final node representations and applying fully connected layers produces an output for the architecture’s global accuracy property.
3. Experiments
The experiments analyze the Neural Predictor in NASBench-101 and then search for high-quality mobile models in the ProxylessNAS search space.
- The experiments first analyze Neural Predictor behavior in the controlled NASBench-101 environment.
- The experiments then search for high-quality mobile models in the ProxylessNAS search space.
- The study covers both a controlled benchmark analysis and a mobile-model search application.
3.1. NASBench-101
NASBench-101 benchmarks architecture search over 423,624 consistently trained cell architectures, using single-run validation accuracy for search and averaged test accuracy for final reporting. The Neural Predictor uses a learned accuracy model to curate candidates and substantially improves search efficiency, although validation rankings do not perfectly predict test performance.
- Benchmark: Search uses one randomly selected validation run per architecture, while the selected model’s test accuracy is averaged across three runs.This setup simulates training each architecture once while retaining a multi-run test measure for reporting.
- Baselines: The oracle selects a model with 95.15% validation accuracy but only 94.08% test accuracy, below the 94.32% global test optimum.Across 100 oracle experiments, mean validation and test accuracies were 95.13% and 94.18%, respectively.
- Baselines: Random search reaches 93.66% test accuracy after training 2,000 models, compared with 94.18% for the oracle, with 0.25% standard deviation.This leaves a substantial gap to the oracle and motivates more efficient candidate selection.
- Neural Predictor: With N = 172, the predictor reaches Regularized Evolution’s 95.06% validation and 94.04% test accuracies 12.40× and 22.83× faster, respectively.The two-stage filter reduces validation-accuracy MSE from 1.95 to 0.66, and the method has small search variance.
- Neural Predictor: The N–K trade-off is constrained in both directions: small N weakens prediction, while large N such as 860 leaves too little evaluation budget.Increasing N improves predictor training data but reduces K under a fixed total compute budget.
3.2. ImageNet Experiments
ImageNet experiments test whether Neural Predictor generalizes to the large ProxylessNAS search space and can identify accurate models under latency constraints. It achieves performance comparable to ProxylessNAS while outperforming random search.
- Experimental setup: The ImageNet experiments target models with inference times around 84 ms on a Pixel-1 phone.The random baseline samples 256 models between 75 ms and 85 ms.
- Search space: The ProxylessNAS search space contains approximately 6.64×10^17 models and requires selecting layer operations and skip connections.The search space uses independent layer choices rather than NASBench-101’s cell-based structure.
- Neural Predictor: Neural Predictor transfers to ProxylessNAS using the same basic pipeline but a single-stage predictor and a modified node representation.The predictor is trained on 119 models, using 79 for training and 40 for validation.
- Predictive performance: The predicted accuracy generalized to unseen architectures, with the sampled models’ trained accuracies exceeding 76%.The predictor evaluated 100K randomly sampled architectures spanning predicted accuracies from 72.94% to 78.45%.
- Neural Predictor: 112,000 latency-constrained architectures were scored, and soft-Pareto selection with J = 6 retained 137 promising models for final training and validation.Soft-Pareto optimality compares each predicted accuracy with the minimum predicted accuracy among the previous six faster models.
- Results: Neural Predictor outperformed random search and performed comparably to ProxylessNAS on validation and test Pareto frontiers.Both methods produced models of similar quality in the ProxylessNAS search space.
4. Discussion and Related Work
The discussion finds Neural Predictor substantially more sample-efficient than Regularized Evolution on NASBench-101 and comparable to ProxylessNAS on ImageNet. Its practical trade-offs depend on implementation complexity and available compute.
- Discussion: 22.83 times greater sample efficiency than Regularized Evolution was observed for Neural Predictor on NASBench-101.Both methods clearly outperformed random search on that benchmark.
- Discussion: Neural Predictor and ProxylessNAS produced models of similar quality in the ProxylessNAS search space.The authors describe Neural Predictor as more straightforward to implement, while a single ProxylessNAS search uses fewer resources.
- Computational trade-offs: Training all N + K = 256 Neural Predictor models required 47.5 times as much compute as a single ProxylessNAS search.With complete parallelization, the Neural Predictor experiment could finish in half the time of a ProxylessNAS search.
- Practical boundary: Method choice depends on search-space implementation complexity, including weight sharing, and the computational resources available.The authors characterize Neural Predictor and ProxylessNAS as complementary approaches.
6. Supplementary Material
The supplementary material details model representations, baselines, training procedures, evaluation conventions, and discovered architectural patterns. It also records implementation-specific choices underlying the reported experiments.
- Ablation study: The NASBench-101 ablation compares GCN and MLP predictors, with the selected GCN achieving the best performance.The simple MLP can also outperform state-of-the-art Regularized Evolution.
- Baselines: Regularized Evolution is reproduced with population size 100, sampling size 10, and mutation probabilities of 1/14 for edges and 1/10 for nodes.The reproduction follows the NASBench-101 paper and released code.
- Evaluation conventions: NASBench-101 test accuracy is reported as the mean over three records rather than a single training run.This mean is intended to represent quality expectations when architectures are retrained by different users.
- Neural Predictor architecture: The Neural Predictor represents architectures with graph convolutions, averages final node representations, and applies fully connected layers to predict accuracy.The final reported architecture uses 18 bidirectional graph convolutional layers and fully connected hidden sizes 512 and 128.
- Training details: The ImageNet predictor uses 119 samples, with 79 training samples and 40 validation samples, and hyperparameters averaged over 10 random splits.The supplementary note explains that one of 120 trained models crashed, leaving 119 samples.
- Discovered architectures: The discovered frontier architectures use cheap early operations or skips and more diverse operations in later layers to form accuracy–inference-time trade-offs.The architectures are represented by operation indices across 22 layers.