Source-linked AI summary
Rethinking Architecture Selection in Differentiable NAS
Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, Cho-Jui Hsieh
TL;DR
The paper asks whether architecture-parameter magnitude reliably identifies operation strength in differentiable NAS, an assumption underlying DARTS selection. It replaces magnitude-based selection with perturbation-based measurement of operation contributions and finds consistently improved architectures from several supernets, while alleviating DARTS robustness issues.
Problem
Differentiable NAS commonly assumes that architecture-parameter magnitude reflects operation strength, but the validity of this selection rule has received little attention.
Method
The paper selects operations by measuring how much discretization perturbs supernet performance, with fine-tuning between edge decisions to maintain convergence.
Results
The perturbation-based method consistently extracts improved architectures from supernets trained by several differentiable NAS methods and improves DARTS test error from 3.00% to 2.61%.
Takeaways & Limitations
Architecture selection is crucial in differentiable NAS, and replacing magnitude-based selection can alleviate several DARTS failure modes without changing the underlying supernet training.
Takeaways & Limitations
The theoretical skip-connection analysis can be extended to search spaces by grouping all non-skip operations into a single operation.
Abstract
from arXiv · showhide
Differentiable Neural Architecture Search is one of the most popular Neural Architecture Search (NAS) methods for its search efficiency and simplicity, accomplished by jointly optimizing the model weight and architecture parameters in a weight-sharing supernet via gradient-based algorithms. At the end of the search phase, the operations with the largest architecture parameters will be selected to form the final architecture, with the implicit assumption that the values of architecture parameters reflect the operation strength. While much has been discussed about the supernet's optimization, the architecture selection process has received little attention. We provide empirical and theoretical analysis to show that the magnitude of architecture parameters does not necessarily indicate how much the operation contributes to the supernet's performance. We propose an alternative perturbation-based architecture selection that directly measures each operation's influence on the supernet. We re-evaluate several differentiable NAS methods with the proposed architecture selection and find that it is able to extract significantly improved architectures from the underlying supernets consistently. Furthermore, we find that several failure modes of DARTS can be greatly alleviated with the proposed selection method, indicating that much of the poor generalization observed in DARTS can be attributed to the failure of magnitude-based architecture selection rather than entirely the optimization of its supernet.
1 INTRODUCTION
Differentiable NAS commonly selects final operations by the largest architecture parameters, but the validity of treating parameter magnitude as operation strength has been underexamined. The paper analyzes this assumption and proposes perturbation-based selection that measures operation contributions directly, improving architectures extracted from existing supernets.
- DARTS jointly optimizes continuous architecture parameters and model weights, then selects operations with the largest parameters to form the final architecture.
- Prior work largely attributes DARTS failures to supernet optimization, while this paper examines the less-studied validity of magnitude-based architecture selection.
- Architecture parameters do not necessarily indicate operation importance: larger values may fail to produce higher validation accuracy after discretization.
- The paper proposes perturbation-based selection, choosing each operation by how much discretization perturbs supernet accuracy and fine-tuning between edge decisions.
- The method consistently extracts significantly improved architectures from DARTS, SDARTS, and SGAS supernets, while alleviating DARTS robustness issues.
2 BACKGROUND AND RELATED WORK
DARTS uses continuous relaxation and bilevel optimization to search efficiently, but its final architecture is still selected by parameter magnitude. Related analyses connect failures to supernet curvature, while this work argues that selection itself can alleviate robustness problems and complements search-space shrinking.
- DARTS represents categorical operation choices with continuous architecture parameters and jointly optimizes them with model weights in a supernet.
- After search, DARTS selects the operation with the largest α_o on each edge to construct the final architecture.
- Prior robustness analyses associate degenerated DARTS performance with high-curvature validation-loss regions in the supernet.
- This paper instead examines architecture selection and shows that a better selection method can alleviate much of DARTS’ robustness issue.
- Unlike progressive shrinking methods that prune operations by α magnitude, the proposed approach selects operations by their contribution to supernet performance.
3 THE PITFALL OF MAGNITUDE-BASED ARCHITECTURE SELECTION IN DARTS
Magnitude-based selection can disagree with an operation’s converged discretization accuracy because DARTS’ architecture parameters reflect optimization dynamics rather than operation contribution alone. Skip connections may therefore dominate α while being a poor selection basis.
- Operation strength is defined as supernet accuracy after discretizing an edge to that operation and fine-tuning until convergence.
- The magnitude of α does not necessarily match relative discretization accuracy, and operations with small α can produce high converged accuracy.
- In S2, discretizing to sep conv 3x3 benefits the supernet more than skip connect on half of the edges.
- 3.2 A CASE STUDY: SKIP CONNECTION: Skip-connection domination can arise reasonably because DARTS’ supernet resembles ResNet and successive edges estimate a shared optimal feature map.
- 3.2 A CASE STUDY: SKIP CONNECTION: Because the edge input can be closer to the optimal feature map than a single convolution output, optimization naturally yields α_skip greater than α_conv.
- 3.2 A CASE STUDY: SKIP CONNECTION: As the supernet improves, the α_skip−α_conv gap grows, turning a reasonable optimization outcome into an inductive bias when α selects architectures.
4 PERTURBATION-BASED ARCHITECTURE SELECTION
The paper replaces magnitude-based operation selection with perturbation-based evaluation that measures each operation’s contribution to supernet performance and progressively discretizes the architecture.
- 4 PERTURBATION-BASED ARCHITECTURE SELECTION: The method directly evaluates operation contribution rather than relying on architecture-parameter magnitude to select the final architecture.It is termed perturbation-based architecture selection, or PT.
- 4.1 EVALUATING THE STRENGTH OF EACH OPERATION: Operation strength is defined by its contribution to supernet performance, measured through validation accuracy after discretization and convergence-restoring fine-tuning.Evaluating every operation this way is accurate but computationally expensive.
- 4.1 EVALUATING THE STRENGTH OF EACH OPERATION: The practical criterion masks each operation individually and selects the operation whose removal causes the largest drop in supernet validation accuracy.Masking perturbs the supernet less than full discretization.
- 4.2 THE COMPLETE ARCHITECTURE SELECTION PROCESS: The algorithm evaluates operations on a randomly selected edge, chooses the best operation, discretizes it, and fine-tunes the remaining supernet for a few epochs.This process repeats until all edges are decided, with cell topology selected similarly.
5 EXPERIMENTAL RESULTS
Experiments apply perturbation-based selection to DARTS and related methods while keeping search settings largely unchanged. The resulting architectures improve test performance, and snapshots of DARTS supernets remain useful when selected with PT.
- 5 EXPERIMENTAL RESULTS: Perturbation-based selection consistently finds better architectures than α-based selection across DARTS and NAS-Bench-201 evaluations.The study also evaluates variants of DARTS and uses identical search and retraining settings where applicable.
- 5.1 RESULTS ON DARTS’ CNN SEARCH SPACE: 3.00% to 2.61%: DARTS+PT reduces DARTS’ CIFAR-10 test error with 0.8 GPU days of search cost.Only the architecture-selection method is changed in this comparison.
- 5.1 RESULTS ON DARTS’ CNN SEARCH SPACE: 2.67% to 2.54% and 2.66% to 2.56%: PT improves the test errors of SDARTS and SGAS, respectively.The best discovered architecture reaches 2.44% test error.
- 5.2 PERFORMANCE ON NAS-BENCH-201 SEARCH SPACE: DARTS supernet snapshots yield stable improvements when PT selects architectures over training trajectories on NAS-Bench-201 datasets.Magnitude-based selection degenerates over time, whereas PT extracts better architectures from the same supernets.
- 5.1 RESULTS ON DARTS’ CNN SEARCH SPACE: Table 2 compares state-of-the-art image classifiers on CIFAR-10, with results accompanied by reported search-cost and averaging annotations.The table caption identifies the dataset and comparison scope.
6 ANALYSIS
Across multiple search spaces and datasets, perturbation-based selection consistently extracts better architectures than magnitude-based selection. It also matches or exceeds the perturbation method when α is fixed uniformly during supernet training.
- Perturbation-based selection consistently extracts meaningful architectures with significantly improved performance across four search spaces and datasets.
- On S2, DARTS degenerates to skip connections, whereas DARTS+PT selects an architecture containing four convolutions.
- On S4, DARTS+PT favors sep conv 3x3 on edges where magnitude-based selection chooses noise.
- DARTS+PT selects operations that produce higher supernet validation accuracy than DARTS+PT-Mag during operation selection on S2.
- Fixing α = 0 during supernet training performs on-par with DARTS+PT and sometimes better, including on NAS-Bench-201.
7 CONCLUSION AND DISCUSSION
The paper reframes differentiable NAS around architecture selection rather than only supernet optimization. Its perturbation-based method measures operation contributions directly and consistently improves architectures extracted from identically trained supernets.
- The paper provides empirical and theoretical evidence that DARTS architecture parameters do not reliably indicate underlying operation strength.
- Perturbation-based selection measures each operation’s strength through its contribution to supernet performance.
- Across several search spaces and datasets, the method consistently extracts improved architectures from supernets trained identically to the base methods.
- Because it does not rely on α to derive the final architecture, the method allows more freedom in supernet training.
A.1.1 DARTS
DARTS represents architecture choices with continuous parameters and selects one operation per edge from the operation pool after search. It also selects two input edges for each node by comparing edge-level maximum α values.
- DARTS selects one operation for each edge in the normal and reduction cells using architecture parameter α.
- The final architecture selects two input edges for each node by comparing the largest α value on every candidate input edge.
- Each edge chooses among seven operations, including skip connection, pooling, separable convolution, and dilated convolution variants.
- The “none” operation determines topology but is not treated as an operation when applying the proposed selection method on DARTS.
A.1.2 SDARTS
The appendix evaluates perturbation-based architecture selection across differentiable NAS methods, search spaces, datasets, and training settings. Results show that operation importance is not reliably captured by α magnitude, while perturbation-based selection remains effective and transfers to ImageNet.
- Architecture selection: The perturbation-based algorithm selects edge operations by validation accuracy after removing each candidate, then fine-tunes the remaining supernet before continuing.It subsequently selects the two input edges per node with the lowest removal-induced validation accuracies and prunes the others.
- Empirical evaluation: Across six randomly seeded DARTS supernets, α magnitude does not necessarily agree with relative discretization accuracy at convergence.The comparison samples one edge from each supernet and evaluates operation strength after discretization and convergence.
- Ablation and robustness: DARTS+PT (fix α) consistently achieves strong NAS-Bench-201 performance without training α, and can outperform DARTS+PT on that benchmark.With α fixed at zero, all operations receive uniform weights during supernet training.
- Ablation and robustness: Gains from fine-tuning beyond five epochs are marginal, supporting the use of five fine-tuning epochs between successive edge decisions.This setting was used because it was empirically sufficient for the supernet to converge again.
- Transferability: On ImageNet in the mobile setting, the derived architecture improves DARTS top1 performance by 1.2%.The evaluation follows DARTS training protocols and hyperparameter settings.