Source-linked AI summary
DARTS: Differentiable Architecture Search
Hanxiao Liu, Karen Simonyan, Yiming Yang
TL;DR
Architecture search can deliver strong models but remains computationally demanding. DARTS relaxes the search space continuously for gradient-based optimization, matching or surpassing non-differentiable methods across vision and language tasks with several-orders-of-magnitude greater efficiency.
Problem
Existing architecture search methods achieve strong performance but remain computationally demanding, motivating more scalable automated design.
Method
DARTS continuously relaxes the architecture search space and uses bilevel gradient-based optimization for convolutional and recurrent architectures.
Results
DARTS matches or outperforms state-of-the-art non-differentiable methods across image classification and language modeling while reducing architecture-discovery costs to a few GPU days.
Takeaways & Limitations
Differentiable architecture search provides a broadly applicable and highly efficient alternative to reinforcement-learning and evolutionary search.
Takeaways & Limitations
The continuous architecture encoding may differ from the discrete architecture ultimately derived, potentially affecting the method's performance.
Abstract
from arXiv · showhide
This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques. Our implementation has been made publicly available to facilitate further research on efficient architecture search algorithms.
1 INTRODUCTION
DARTS addresses the computational cost of neural architecture search by continuously relaxing the architecture space and optimizing it with gradient descent. It applies to convolutional and recurrent networks, achieving competitive results while reducing architecture-discovery costs to a few GPU days.
- Motivation: Prior state-of-the-art searches required 2000 GPU days of reinforcement learning or 3150 GPU days of evolution for CIFAR-10 and ImageNet.These computational demands motivate more efficient search methods.
- Method: DARTS replaces discrete architecture search with a continuous relaxation optimized by gradient descent on validation performance.The method uses bilevel optimization and applies to both convolutional and recurrent architectures.
- Method: DARTS learns high-performance building blocks with complex graph topologies in a rich search space spanning convolutional and recurrent networks.Unlike prior methods focused on specific architectural aspects, DARTS is not restricted to one architecture family.
- Results: 2.76 ± 0.09% test error on CIFAR-10 is achieved by a 3.3M-parameter convolutional cell, competitively with regularized evolution using three orders of magnitude more computation.The same cell achieves 26.7% top-1 error on ImageNet in the mobile setting.
- Results: Gradient-based architecture search achieves highly competitive CIFAR-10 results and outperforms the state of the art on PTB.DARTS reduces architecture-discovery cost to a few GPU days, and architectures learned on CIFAR-10 and PTB transfer to ImageNet and WikiText-2.
2 DIFFERENTIABLE ARCHITECTURE SEARCH
DARTS searches over computation cells by continuously relaxing edge operations and jointly optimizing architecture variables with network weights. It makes bilevel gradient-based search computationally feasible through one-step approximation and finite-difference gradients, then discretizes the learned architecture.
- Search space: DARTS searches for a computation cell that can be stacked into convolutional networks or recursively connected into recurrent networks.The cell is a directed acyclic graph with ordered nodes, two input nodes, and one output node.
- Search space: Each cell edge selects an operation, including a zero operation for absent connections, so learning the cell reduces to learning its edge operations.Intermediate nodes are computed from their predecessors, and the candidate operations include convolution, max pooling, and zero.
- Continuous relaxation: DARTS replaces each categorical edge choice with a softmax mixture of candidate operations parameterized by continuous architecture variables α, then selects the most likely operation to derive a discrete architecture.The architecture encoding α contains the operation mixing weights for each node pair.
- Bilevel optimization: The search is a bilevel optimization in which α minimizes validation loss while weights w minimize training loss for the selected architecture.DARTS optimizes validation loss with gradient descent rather than treating validation performance as a reinforcement-learning reward or evolutionary fitness.
- Efficient approximation: A one-step inner optimization approximates the exact architecture gradient, while finite differences reduce complexity from O(|α||w|) to O(|α| + |w|).The finite-difference procedure requires two forward passes for weights and two backward passes for α.
- Efficient approximation: The first-order approximation sets ξ = 0 and is faster but empirically performs worse, while the iterative algorithm can reach a fixed point with a suitable ξ despite lacking known convergence guarantees.With momentum enabled for weight optimization, the one-step unrolled objective is modified accordingly and the analysis still applies.
3 EXPERIMENTS AND RESULTS
Experiments use a two-stage process to search and evaluate architectures, showing that DARTS discovers competitive CIFAR-10 and PTB models with substantially lower search cost and transferable cells for ImageNet and WikiText-2.
- Experimental procedure: DARTS searches for cells using validation performance, then trains the selected architecture from scratch and evaluates it on the test set.Architecture selection uses four random seeds and short retraining runs of 100 epochs on CIFAR-10 or 300 epochs on PTB; test data is excluded from search and selection.
- PTB results: 55.7 test perplexity on PTB matched the state-of-the-art mixture-of-softmaxes model and surpassed manually or automatically discovered alternatives, including a tuned LSTM.The result demonstrates the value of architecture search for recurrent models.
- Search comparison: DARTS improved over random search with comparable or lower search cost, achieving 2.76 ± 0.09 versus 3.29 ± 0.15 on CIFAR-10 and 55.7 versus 59.4 on PTB.The comparison is reported for both convolutional and recurrent models.
- Transferability: The CIFAR-10 cell transferred to ImageNet with competitive state-of-the-art performance while using three orders of magnitude fewer computation resources.The PTB cell also transferred to WikiText-2 better than ENAS, although transfer results were weaker than on PTB.
4 CONCLUSION
DARTS efficiently searches convolutional and recurrent architectures in a continuous space, matching or outperforming state-of-the-art non-differentiable methods across image classification and language modeling. The authors identify continuous–discrete architecture discrepancies as a limitation and suggest temperature annealing and performance-aware derivation as future directions.
- 4 CONCLUSION: DARTS searches convolutional and recurrent networks in a continuous space, matching or outperforming state-of-the-art non-differentiable methods on image classification and language modeling.The method achieves this with a remarkable efficiency improvement by several orders of magnitude.
- 4 CONCLUSION: The continuous architecture encoding may differ from the derived discrete architecture, potentially harming DARTS’s reliability.The authors suggest annealing the softmax temperature with a suitable schedule to enforce one-hot selection.
- 4 CONCLUSION: Performance-aware architecture derivation schemes based on shared parameters learned during search are proposed as another direction for improving DARTS.
A EXPERIMENTAL DETAILS … A.1.2 PTB
The experimental details describe implementation choices for DARTS architecture search on CIFAR-10 and PTB, including normalization, resource-constrained search configurations, parameter sharing, and optimization settings.
- A.1.1 CIFAR-10: During CIFAR-10 search, batch normalization uses batch-specific statistics instead of global moving averages because the architecture varies throughout search.Learnable affine parameters are disabled to prevent candidate operations from rescaling outputs.
- A.1.1 CIFAR-10: CIFAR-10 search holds out half of the training data for validation and trains an 8-cell network for 50 epochs.The network uses batch size 64, 16 initial channels, and is sized to fit on one GPU.
- A.1.1 CIFAR-10: CIFAR-10 network weights are optimized with momentum SGD using initial learning rate ηw = 0.025 and cosine annealing without restart.The supplied passage states that the learning rate is annealed down to zero.
- A.1.2 PTB: For PTB architecture search, both embedding and hidden sizes are set to 300, with incoming linear transformations sharing parameters across operations connected to each node.All shared transformations have shape 300 × 300, and sharing enables a single-GPU search through memory savings and faster computation.
- A.1.2 PTB: PTB parameter tying allows the continuous architecture to focus on one predecessor while masking away the others.This follows from the algorithm’s option to select one predecessor among incoming operations.
- A.1.2 PTB: 6 hours is the reported duration for PTB architecture search on a single GPU.The architecture parameters α are initialized as zeros, with initial learning rate ηα = 3 × 10−3, momentum β = (0.9, 0.999), and weight decay 10−3.
A.2 ARCHITECTURE EVALUATION … A.2.4 WIKITEXT-2
The architecture evaluation uses task-specific training configurations across CIFAR-10, PTB, ImageNet, and WikiText-2, with explicit controls for model size, fairness, computational cost, and optimization. CIFAR-10 comparisons use a common training framework, while the other tasks specify recurrent, mobile-image, and language-model settings.
- A.2.1 CIFAR-10: CIFAR-10 trains a large 20-cell network for 600 epochs with batch size 96 and 36 initial channels, targeting approximately 3M parameters.The increased channel count is intended to make model size comparable with literature baselines.
- A.2.1 CIFAR-10: CIFAR-10 training includes cutout and path dropout as additional enhancements, while retaining the architecture-search hyperparameters otherwise.The passage situates these enhancements alongside practices from existing works.
- A.2.1 CIFAR-10: NASNet-A and AmoebaNet-A cells are evaluated within the same training framework and settings as the authors’ cells to avoid implementation and training discrepancies.The comparison specifically addresses differences such as batch sizes.
- A.2.1 CIFAR-10: The experiments were performed using NVIDIA GTX 1080Ti GPUs.This hardware note applies to all reported experiments.
- A.2.2 PTB: PTB trains a single-layer recurrent network with the discovered cell to convergence using batch size 64, ASGD, learning rate ηw = 20, and weight decay 8 × 10−7.Training begins with SGD and triggers ASGD using the protocol of Yang et al. (2018) and Merity et al. (2018).
- A.2.3 IMAGENET: ImageNet evaluation uses 224×224 inputs and restricts the model to fewer than 600M multiply-add operations in the mobile setting.The network is trained for 250 epochs with batch size 128 and initial SGD learning rate 0.1, decayed by 0.97 each epoch.
- A.2.4 WIKITEXT-2: WikiText-2 uses embedding and hidden sizes of 700, weight decay 5×10−7, and hidden-node variational dropout 0.15, with other hyperparameters inherited from PTB.The passage defines the WikiText-2 configuration by adapting the PTB setup.
B SEARCH WITH INCREASED DEPTH
Increasing the CIFAR-10 search network from 8 to 20 cells doubled search cost, yet produced a cell with slightly worse test error than the shallower network.
- Search with increased depth: Doubling the stack depth from 8 to 20 cells doubled search cost and yielded 2.88 ± 0.09% test error.The initial channel count was reduced from 16 to 6 because of a single-GPU memory budget, while other hyperparameters remained unchanged.
- Search with increased depth: The deeper network’s 2.88 ± 0.09% test error was slightly worse than the shallower network’s 2.76 ± 0.09%.Both results were obtained on CIFAR-10.
C COMPLEXITY ANALYSIS
The search space contains approximately 10^18 architectures when jointly learning normal and reduction cells, exceeding PNAS’s single-cell search space. Before final discretization, the relaxed continuous search space is even larger, with approximately 4 × 10^12 configurations per cell.
- Discrete search space: Approximately 10^18 architectures arise from jointly learning normal and reduction cells, exceeding PNAS’s 5.6 × 10^14 architectures for a single cell type.The discretized search space contains approximately 10^9 possible DAGs before accounting for graph isomorphism.
- Continuous search space: Each relaxed cell has 14 learnable edges, yielding approximately 4 × 10^12 possible configurations before final top-2 predecessor discretization.The 14 edges come from 2 + 3 + 4 + 5 connections across the fully connected graph, with a zero operation included as a possible configuration.