Source-linked AI summary
NSGANetV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Search
Zhichao Lu, Kalyanmoy Deb, Erik Goodman, Wolfgang Banzhaf, Vishnu Naresh Boddeti
TL;DR
NAS must efficiently optimize both network architectures and their weights, especially when objectives compete and datasets vary. The paper proposes MSuNAS, using online architecture-level and supernet-based weight-level surrogates, and reports competitive or improved models across standard and non-standard datasets with substantially greater search efficiency.
Problem
NAS must jointly find an architecture and its associated weights, but costly weight optimization and extensive architecture sampling hinder efficient multi-objective search.
Method
MSuNAS combines an online upper-level surrogate focused near the current trade-off front with a supernet-based lower-level surrogate for weight learning.
Results
NSGANetV2 matches state-of-the-art performance on CIFAR-10, CIFAR-100, and ImageNet and improves state-of-the-art results on STL-10 and Flowers102 under mobile settings.
Takeaways & Limitations
The reported results demonstrate MSuNAS's utility for task-specific models across datasets with different scales, class structures, and numbers of objectives.
Abstract
from arXiv · showhide
In this paper, we propose an efficient NAS algorithm for generating task-specific models that are competitive under multiple competing objectives. It comprises of two surrogates, one at the architecture level to improve sample efficiency and one at the weights level, through a supernet, to improve gradient descent training efficiency. On standard benchmark datasets (C10, C100, ImageNet), the resulting models, dubbed NSGANetV2, either match or outperform models from existing approaches with the search being orders of magnitude more sample efficient. Furthermore, we demonstrate the effectiveness and versatility of the proposed method on six diverse non-standard datasets, e.g. STL-10, Flowers102, Oxford Pets, FGVC Aircrafts etc. In all cases, NSGANetV2s improve the state-of-the-art (under mobile setting), suggesting that NAS can be a viable alternative to conventional transfer learning approaches in handling diverse scenarios such as small-scale or fine-grained datasets. Code is available at https://github.com/mikelzc1990/nsganetv2
1 Introduction
MSuNAS addresses the two computational bottlenecks of multi-objective NAS with surrogate models at both optimization levels, producing task-specific architectures efficiently across standard and diverse datasets.
- Motivation: NAS jointly optimizes architecture and weights, but costly lower-level training and discrete, multi-objective architecture search make conventional optimization prohibitive.The paper identifies both bottlenecks as targets for simultaneous mitigation.
- Approach: MSuNAS uses explicit surrogates at both levels: an online architecture surrogate improves sample efficiency, while a supernet-based surrogate improves weight-learning efficiency.The upper surrogate focuses on architectures near the current trade-off front, while the lower surrogate initializes fine-tuning through a supernet.
- Scope: MSuNAS designs custom architectures for multiple competing objectives and supports large-scale, small-scale, multi-class, and fine-grained datasets.The method is evaluated on ImageNet, CIFAR-10, CIFAR-100, and six non-standard datasets.
- Results: Under mobile settings of ≤600M MAdds, MSuNAS achieves state-of-the-art performance across the reported dataset scenarios.The cited scenarios include standard datasets and non-standard datasets such as CINIC-10, STL-10, and Oxford Flowers102.
2 Related Work
Prior NAS work improves lower-level training efficiency through weight sharing and related proxies, while upper-level surrogates and multi-objective methods retain important efficiency or preference limitations.
- Lower-Level Surrogates: Weight sharing, proxy models, and coarse training reduce lower-level costs, but upper-level search may still require many architecture evaluations.The cited approaches mitigate weight optimization without fully resolving architecture-sampling demands.
- Upper-Level Surrogates: Offline upper-level surrogates require large training samples, with PNAS using 16K samples and incurring substantial surrogate-construction cost.The passage reports 16K samples, approximately two GPU-days, and twice DARTS's search cost for constructing the surrogate.
- Upper-Level Surrogates: MSuNAS trains its upper-level surrogate online using samples near the current trade-off front rather than uniformly sampling the entire search space.This restricts interpolation to models close to the current Pareto front and improves sample efficiency.
- Multi-Objective NAS: Scalarized multi-objective NAS methods combine objectives with a predefined preference weighting, whereas population-based methods approximate the Pareto-efficient frontier simultaneously.MSuNAS belongs to the population-based category and uses surrogate models to reduce computational overhead.
3 Proposed Approach
MSuNAS formulates NAS as bilevel, multi-objective optimization and uses surrogates at both levels to reduce weight-training and architecture-sampling costs. Its search combines adaptive accuracy prediction, Pareto-based candidate selection, and supernet-initialized fine-tuning.
- Problem formulation: NAS jointly searches architecture variables and associated weights under multiple objectives, including predictive performance and efficiency measures.Architecture-only objectives include parameters, floating-point operations, and latency.
- Search space: The search space decomposes CNNs into five sequential blocks and searches depth, width, kernel size, and input resolution.Each block contains two to four layers, with controlled stride changes as feature-map size decreases.
- Two-level surrogates: MSuNAS uses an upper-level surrogate to predict architecture accuracy and a lower-level supernet surrogate to initialize weight fine-tuning.The two surrogates target architecture sample efficiency and the cost of optimizing candidate weights, respectively.
- Candidate selection: NSGA-II generates non-dominated architectures, after which a diverse subset is evaluated and added to the archive for subsequent surrogate updates.The procedure reports non-dominated architectures from the final archive.
- Accuracy prediction: Adaptive Switching selects among MLP, CART, RBF, and GP accuracy predictors using cross-validation because no single model is consistently best across datasets.The selected predictor is optimized jointly with user-specified objectives through NSGA-II.
4 Experiments and Results
Experiments evaluate surrogate prediction, search efficiency, and resulting models on CIFAR-10, CIFAR-100, and ImageNet. MSuNAS improves search efficiency while NSGANetV2 delivers competitive or stronger accuracy-efficiency trade-offs across standard benchmarks.
- Surrogate predictors: Adaptive Switching outperforms any individual surrogate model in prediction performance across the evaluated datasets.The comparison uses Spearman rank-order correlation between predicted and true accuracy, with 10-fold cross-validation selecting the predictor each iteration.
- Search efficiency: 20x faster than RL- or EA-based methods, MSuNAS remains at least 3x faster than PNAS in single-objective search.The speedup compares architectures evaluated and training epochs per sampled architecture.
- Bi-objective search: 350 architectures are evaluated in the bi-objective MSuNAS search, with cumulative hypervolume compared against NSGANet and random search over five runs.The comparison covers ImageNet and CIFAR-10 and reports variation using standard deviations.
- ImageNet: 2.2% more accurate than MobileNetV3 at equivalent #MAdds and latencies, NSGANetV2-s demonstrates a stronger ImageNet accuracy-efficiency result.NSGANetV2-xl reaches 80.4% Top-1 accuracy under 600M MAdds and is 1.2x more efficient than EfficientNet-B1.
- CIFAR datasets: NSGANetV2 dominates previous models on CIFAR-10 across predictive performance and computational efficiency, including directly searched and transferred models.The comparison includes NASNet-A, PNASNet-5, NSGANet, EfficientNet, MobileNetV3, and MixNet.
5 Scalability of MSuNAS
MSuNAS is evaluated beyond standard benchmarks by searching directly on diverse datasets and by handling scalarized, bi-objective, and many-objective formulations. The results support task-specific architecture search under varied data regimes and competing efficiency objectives.
- Non-standard datasets: Existing NAS methods rarely test search beyond ImageNet, CIFAR-10, and CIFAR-100, instead transferring architectures to new datasets.The paper evaluates direct searching as an alternative experimental setting for non-standard datasets.
- Search cost: Less than one day on 8 GPU cards is required for one search on each non-standard dataset.The reported runtime uses the paper’s stated dataset-search setup.
- Non-standard datasets: NSGANetV2 significantly outperforms transferred models on CINIC-10, STL-10, and Oxford Flowers102 when searched directly on each dataset.On Oxford Flowers102, it achieves better accuracy than EfficientNet-B3 while using 1.4B fewer MAdds.
- Single-objective formulation: 79.56% accuracy with 596M MAdds is obtained on ImageNet using scalarized optimization targeting 600M MAdds.The architecture is obtained without further fine-tuning in the reported evaluation.
- Many-objective formulation: Five-objective search produces a Pareto surface over accuracy, #Params, #MAdds, CPU latency, and GPU latency.Accuracy is highly correlated with #MAdds and both latency measures, but not with #Params to the same extent.
6 Conclusion
MSuNAS combines upper- and lower-level surrogates to efficiently design task-specific models under multiple competing objectives. It matches state-of-the-art results on standard datasets and improves them on STL-10 and Flowers102 under mobile settings.
- MSuNAS uses architecture-level online surrogate modeling to improve search sample efficiency and a supernet-based surrogate to improve weights-learning efficiency through fine-tuning.
- On CIFAR-10, CIFAR-100, and ImageNet, NSGANetV2 matches the state-of-the-art with a search cost of one day.
- On STL-10 and Flowers102 under mobile settings, NSGANetV2 improves beyond the state-of-the-art.
- The appendix covers upper-level surrogate analysis, architectural design insights, objective transfer, and visualizations across six searched datasets.
A Correlation Between Search Performance and Surrogate Model
MSuNAS evaluates surrogate models by their rank-order correlation with true architecture performance and by search hypervolume. Focusing online modeling near the Pareto frontier yields substantially stronger correlation than ProgressiveNas, with correlation associated with search performance.
- A useful surrogate model should combine high rank-order correlation with high sample efficiency, requiring few fully trained and evaluated architectures.
- MSuNAS evaluates MLP, CART, RBF, and GP surrogates on ImageNet using Kendall’s Tau and accumulated hypervolume across architectures evaluated during search.
- MSuNAS achieves approximately 0.9 rank-order correlation, compared with 0.476 for ProgressiveNas, by fitting surrogates near the Pareto frontier.
- The reported analysis observes a positive relationship between surrogate rank-order correlation and search performance.
B.1 Mining for Insights
MSuNAS’s non-dominated architectures reveal design patterns across efficiency objectives. MAdds, CPU latency, and GPU latency produce similar efficient architectures, whereas parameter count differs because it is resolution-agnostic.
- MSuNAS mines non-dominated architectures after optimizing predictive performance with MAdds, Params, CPU latency, or GPU latency.
- The analysis shows that a single MSuNAS run can provide architectural choices that practitioners inspect after optimization.
- Efficient architectures under MAdds, CPU latency, and GPU latency are similar, indicating positive correlation among those objectives.
- Parameter-count optimization produces different architectural choices because Params is agnostic to image resolution.
B.2 Transfer Across Objectives
MSuNAS can transfer architectural insights from one objective pair to initialize search for another. The resulting efficiency gain depends on how closely the source and target objectives are related.
- Objective transfer reuses architectures and associated weights from models optimized for one objective pair to search for architectures optimized for another pair.
- The target search optimizes predictive performance and CPU latency, comparing random initialization with initialization from architectures sampled using predictive performance and MAdds.
- Search initialized from architectures derived from related objectives significantly improves hypervolume performance over five runs.
- Objective-transfer efficiency gains are directly proportional to source-target objective correlation, and unrelated objectives may provide no advantage over random initialization.
C Evolved Architectures
NSGANetV2 searches directly on each target dataset to evolve architectures that trade predictive performance against MAdds. The resulting architectures differ across datasets, reflecting dataset-specific efficiency requirements.
- NSGANetV2 architectures are found by simultaneously maximizing predictive performance and minimizing MAdds.
- Different datasets require different architectures to achieve efficient trade-offs between MAdds and performance.
- All architectures share the same stem layers and contain five blocks, with stride-2 first layers in blocks 1, 2, 3, and 5.
- Directly searching on the target dataset enables these dataset-specific architectures.