Source-linked AI summary
CARS: Continuous Evolution for Efficient Neural Architecture Search
Zhaohui Yang, Yunhe Wang, Xinghao Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, Chang Xu
TL;DR
NAS methods can be computationally expensive, while efficient differentiable methods may produce insufficient architectural variety. CARS uses continuous evolutionary search with a shared SuperNet, inherited populations, Pareto selection, and protection for larger models. The framework produces Pareto-front networks efficiently and reports superior results on benchmark datasets, including ImageNet.
Problem
Existing NAS methods face high search costs, while gradient-based approaches can suffer from limited architectural variety and evolutionary approaches evaluate candidates expensively.
Method
CARS alternates parameter and architecture optimization using a shared SuperNet, inherited populations, genetic evolution, Pareto selection, and pNSGA-III protection for larger models.
Results
CARS efficiently provides several architectures on the Pareto front and reports superior performance over state-of-the-art methods on benchmark datasets.
Takeaways & Limitations
CARS provides a set of transferable architectures spanning different model sizes and performances through one-time, parameter-sharing search.
Takeaways & Limitations
The search procedure must address the small model trap, in which faster convergence and fluctuating accuracy can eliminate larger architectures.
Abstract
from arXiv · showhide
Searching techniques in most of existing neural architecture search (NAS) algorithms are mainly dominated by differentiable methods for the efficiency reason. In contrast, we develop an efficient continuous evolutionary approach for searching neural networks. Architectures in the population that share parameters within one SuperNet in the latest generation will be tuned over the training dataset with a few epochs. The searching in the next evolution generation will directly inherit both the SuperNet and the population, which accelerates the optimal network generation. The non-dominated sorting strategy is further applied to preserve only results on the Pareto front for accurately updating the SuperNet. Several neural networks with different model sizes and performances will be produced after the continuous search with only 0.4 GPU days. As a result, our framework provides a series of networks with the number of parameters ranging from 3.7M to 5.1M under mobile settings. These networks surpass those produced by the state-of-the-art methods on the benchmark ImageNet dataset.
1. Introduction
NAS automates architecture design but existing approaches can be computationally expensive or lack architectural variety. CARS addresses this with continuous evolution, parameter sharing, Pareto selection, and protection for larger models.
- Over-parameterized networks can deliver strong performance but consume substantial computational resources, while manual architecture design depends heavily on expert knowledge and repeated experimentation.
- Figure 1 evaluates CARS-Lat using validation performance and HUAWEI P30 Pro latency during search, then reports Top-1 accuracy on ILSVRC2012.
- NAS automates architecture design, yet existing methods often require extensive computational resources to find architectures with acceptable performance.
- EA- and RL-based methods are inefficient because candidate architectures are trained separately for many epochs, whereas gradient-based methods trade efficiency for reduced architectural variety.
- Evolutionary NAS is costly because each individual is independently evaluated, and weight sharing may require SuperNet training to compensate for poor architectures.
- CARS continuously inherits the updated SuperNet and Pareto-selected population across generations, while pNSGA-III protects larger models from the small model trap.
2. Related Works
Related NAS work separates parameter optimization from architecture optimization and includes independent, shared-parameter, RL, EA, gradient-based, and multi-objective strategies. CARS alternates optimization stages while sharing parameters to improve search efficiency.
- Gradient-based NAS separates network parameter optimization from architecture optimization, with the latter learning patterns of accurate architectures.
- Independent optimization trains each architecture separately, whereas one-shot methods share parameters within a SuperNet to reduce training requirements.
- RL-based methods train an architecture controller using generated architectures' performance as rewards, while EA-based methods evolve populations using validation accuracy as fitness.
- Multi-objective NAS seeks Pareto-front architectures because accuracy, parameter count, FLOPs, energy, and latency cannot generally be optimized by one architecture simultaneously.
- CARS alternates architecture and parameter optimization while sharing parameters across architectures, making the search stage more efficient than separate optimization.
3. Approach
CARS is a continuous evolutionary NAS framework that alternates shared SuperNet parameter optimization with multi-objective architecture evolution. Its pNSGA-III strategy protects larger and middle-sized candidates from the small-model trap while retaining diverse Pareto-optimal architectures.
- CARS pipeline: CARS alternates parameter optimization and architecture optimization to search neural architectures continuously.The pipeline also introduces parameter warm-up before these two procedures.
- Parameter optimization: A SuperNet shares parameters across sampled architectures, reducing the computational complexity of training each architecture separately.Each architecture is represented by binary connection parameters that mask the shared full-precision parameters.
- Parameter optimization: Shared parameters are updated by accumulating gradients from mini-batches of architectures, whose averaged gradient provides an unbiased approximation of the population gradient.Using B architectures with B < P reduces update time while balancing efficiency and accuracy through the mini-batch size.
- Architecture optimization: Non-dominated sorting selects architectures that are no worse across objectives and better on at least one, then uses them to update the SuperNet.The objectives can include model measurements such as parameters, FLOPs, latency, energy, and accuracy.
- Architecture optimization: NSGA-III can eliminate larger models because incompletely optimized accuracies let smaller models dominate them during evolution.This small-model trap occurs because current-generation accuracy may not reflect an architecture’s eventual performance.
- Architecture optimization: pNSGA-III merges Pareto rankings based on model size with accuracy and on model size with accuracy-increase speed, preserving larger and accurate middle-sized models.The resulting population protects larger models and covers a broad range of model sizes, unlike conventional NSGA-III.
4. Experiments
Experiments evaluate CARS across CIFAR-10 and ILSVRC2012, including device-agnostic and device-aware objectives. Results examine the small model trap, compare evolutionary selection strategies, and show Pareto-efficient architectures with competitive or superior performance.
- Experimental setup: CARS searches architectures across multiple model sizes and performances using CIFAR-10, with comparisons framed around performance and model size.The search maintains 128 architectures and uses pNSGA-III after network parameters are updated for ten epochs.
- Small model trap: After 600 epochs, CIFAR-10 accuracy rises from 96.87% to 97.20% as model size increases from 2.1M to 3.6M parameters.The early training curves can instead favor smaller models, motivating the small model trap analysis.
- Small model trap: pNSGA-III protects larger architectures that converge more slowly, whereas NSGA-III eliminates them during early architecture optimization.Maintaining larger models preserves candidates with potential to improve later and supports searching across varied computing resources.
- CIFAR-10 results: 2.4M–3.6M parameters are covered by CARS models on CIFAR-10, whose performances are on par with state-of-the-art methods.Using NSGA-III instead produces approximately 2.4M-parameter architectures without larger models and with relatively poor performance.
- CIFAR-10 results: CARS-H achieves 2.66% test error versus 2.76% for DARTS (second-order) at the same 3.3M parameters.The CARS-H reduction block preserves more useful information, while its smaller normal block avoids unnecessary computations.
- ILSVRC2012 results: 3.7M–5.1M model sizes and 430–590 MFLOPs are covered after transfer to ILSVRC2012, while device-aware searches cover 40ms–90ms runtime latency.CARS-I surpasses PNAS by 1% Top-1 accuracy at the same number of parameters and approximate FLOPs; CARS-G exceeds DARTS by 0.9% at the same parameter count.
5. Conclusion
CARS efficiently produces multiple architectures on the Pareto front, outperforming state-of-the-art methods in model size/latency and accuracy.
- CARS uses continuous evolution to reuse architectures and parameters from the latest generation, reducing the inefficiency of separately training each candidate.
- pNSGA-III selects architectures for updating the SuperNet and preserves several Pareto-front models with different trade-offs.
- Experiments on benchmark datasets show that CARS provides several architectures on the Pareto front.
- The searched models are superior to state-of-the-art methods in model size/latency and accuracy.