PaperlayerView exact revision on arXiv

Source-linked AI summary

Stabilizing Differentiable Architecture Search via Perturbation-based Regularization

Xiangning Chen, Cho-Jui Hsieh

arXiv:2002.05283v3cs.LGcs.CVstat.ML

TL;DR

DARTS is computationally efficient but can produce unstable discrete architectures because projection from a continuous mixture may encounter a precipitous validation-loss landscape. The paper proposes SmoothDARTS, which uses random smoothing or adversarial perturbations to smooth that landscape, and reports improved stability and performance while linking the effect to implicit Hessian-norm regularization.

  • Problem

    DARTS can show improving mixture-architecture validation accuracy while the projected discrete architecture suffers a large and unstable performance drop.

  • Method

    SmoothDARTS regularizes architecture search by optimizing network weights under either expected-loss random perturbations or worst-case adversarial perturbations of architecture weights.

  • Results

    SmoothDARTS consistently improves DARTS-based methods and matches or improves state-of-the-art results across CIFAR-10, ImageNet, and Penn Treebank search spaces.

  • Takeaways & Limitations

    The methods produce smoother landscapes and improved stability, with the paper mathematically attributing this effect to implicit regularization of the validation-loss Hessian norm.

  • Takeaways & Limitations

    A high-dimensional perturbation space limits direct gradient-based updating of architecture weights because only a subset of perturbation directions can be sampled each iteration.

Abstract

from arXiv · show

Differentiable architecture search (DARTS) is a prevailing NAS solution to identify architectures. Based on the continuous relaxation of the architecture space, DARTS learns a differentiable architecture weight and largely reduces the search cost. However, its stability has been challenged for yielding deteriorating architectures as the search proceeds. We find that the precipitous validation loss landscape, which leads to a dramatic performance drop when distilling the final architecture, is an essential factor that causes instability. Based on this observation, we propose a perturbation-based regularization - SmoothDARTS (SDARTS), to smooth the loss landscape and improve the generalizability of DARTS-based methods. In particular, our new formulations stabilize DARTS-based methods by either random smoothing or adversarial attack. The search trajectory on NAS-Bench-1Shot1 demonstrates the effectiveness of our approach and due to the improved stability, we achieve performance gain across various search spaces on 4 datasets. Furthermore, we mathematically show that SDARTS implicitly regularizes the Hessian norm of the validation loss, which accounts for a smoother loss landscape and improved performance.

1. Introduction

DARTS reduces NAS search cost through continuous architecture relaxation, but its validation gains can yield unstable, poorly generalizing discrete architectures. SmoothDARTS addresses this instability by smoothing the architecture-weight landscape through random or adversarial perturbations, with theoretical and empirical support.

  • Motivation: DARTS learns differentiable architecture weights in a continuous mixture, reducing the cost of searching a discrete architecture space.Its formulation builds on weight sharing and continuous relaxation.
  • Motivation: Validation accuracy can keep improving while the derived architecture collapses during evaluation, producing distorted structures and unstable search.Reported distortions include skip-connection dominance and a preference for wide, shallow networks.
  • Motivation: The projection from a continuous mixture to a discrete architecture can cause a large performance drop because sharp regions are highly sensitive to architecture-weight perturbations.The associated network weights may work only for the current architecture weight, limiting exploration.
  • Contribution: SmoothDARTS optimizes weights for nearby architecture configurations using either expected-loss random smoothing or worst-case adversarial perturbations.SDARTS-RS uses neighborhood-averaged loss, while SDARTS-ADV minimizes worst-case neighborhood loss.
  • Contribution: The proposed regularization implicitly minimizes the validation-loss Hessian norm, yielding smoother landscapes and improved stability across settings.The paper connects the Hessian regularization to the discretization-related performance drop.

2. Background and Related Work

DARTS relaxes discrete cell-operation choices into continuous architecture weights and alternates updates of architecture and network parameters. Related work targets search efficiency and stability, while SmoothDARTS applies perturbation robustness to stabilize NAS.

  • 2.1. Differentiable Architecture Search: DARTS represents each cell edge with a mixed operation and relaxes discrete operation selection into a continuous architecture-weight vector.The architecture weight A concatenates the operation-weight vectors for all edges.
  • 2.1. Differentiable Architecture Search: DARTS uses bi-level optimization, alternately updating architecture weights and network weights while approximating the inner optimum with current or one-step weights.This enables gradient-based optimization of the relaxed architecture.
  • 2.1. Differentiable Architecture Search: DARTS variants reduce memory or search-evaluation mismatch, and the proposed regularization can be applied to them with consistent improvements.PC-DARTS evaluates a random proportion of channels, while P-DARTS progressively increases mixture depth.
  • Stabilize DARTS: Prior stabilization methods include early stopping, partial channel connections, ScheduledDropPath, and L2 regularization on network weights.Early stopping uses a criterion based on the dominant architecture-weight Hessian eigenvalue.
  • 2.2. Adversarial Robustness: Random smoothing and adversarial training motivate treating architecture-weight perturbations as a robustness problem during NAS.The paper presents this as the basis for stabilizing architecture search.

3. Proposed method

SDARTS stabilizes DARTS by training architecture weights against local perturbations, using either expected-loss random smoothing or worst-case adversarial loss. These formulations target smoother validation landscapes and improve stability during projection to discrete architectures.

  • Motivation: DARTS can produce unstable final architectures because projecting continuous architecture weights to discrete choices may cause a large validation-error increase.The continuous mixture architecture can improve during search while the projected architecture remains unstable or deteriorates.
  • Proposed formulation: SDARTS redefines the optimized weight as one that performs well throughout a neighborhood of the current architecture weight.This shifts optimization toward architecture–weight pairs with smoother loss landscapes.
  • Random smoothing: SDARTS-RS minimizes expected training loss under small random perturbations, using sampled perturbations that can be optimized directly with SGD.Uniform perturbations are used for simplicity, while other distributions such as Gaussian perturbations can also be used.
  • Adversarial smoothing: SDARTS-ADV minimizes worst-case training loss under bounded perturbations, making it stronger than the expected-loss requirement of SDARTS-RS.The perturbation is computed with multi-step projected gradient descent, with projection onto the selected norm ball.
  • Effect and optimization: SDARTS-RS adds only one line of code without overhead and removes the sharp cone, maintaining high validation accuracy under architecture perturbations.SDARTS-ADV produces an even smoother landscape but requires solving a min-max optimization problem before updating the shared weight.

4. Implicit Regularization on Hessian Matrix

The paper links projection-induced performance loss to the validation-loss Hessian with respect to architecture weights. It argues that both random smoothing and adversarial smoothing implicitly reduce Hessian-related curvature, whereas explicit Hessian control is costly and less effective in the reported comparison.

  • Hessian and solution quality: The dominant Hessian eigenvalue is correlated with DARTS solution quality, while standard DARTS training typically produces increasing Hessian norms and deteriorating test performance.Both SDARTS-RS and SDARTS-ADV consistently reduce Hessian norms during training.
  • Projection analysis: A Taylor-expansion analysis bounds projection-induced performance drop using the average Hessian and the squared distance between continuous and discrete architecture weights.The bound assumes a locally stable Hessian and fixed network weights; fine-tuning can make the projected validation loss smaller.
  • Explicit regularization: Explicit Hessian regularization estimates curvature with finite differences, but it requires substantial additional memory and performs worse than SDARTS-RS and SDARTS-ADV in Figure 3.The method may also miss important directions because the architecture-weight space is high-dimensional.
  • Random-smoothing analysis: For SDARTS-RS, taking expectations over zero-mean perturbations cancels first-order and off-diagonal terms, so updating weights implicitly controls the validation-loss Hessian trace norm.When the Hessian is close to positive semidefinite, this approximately regularizes its positive eigenvalues.
  • Adversarial analysis: SDARTS-ADV approximately minimizes Hessian norm under locally constant curvature, with the exact second-order term depending on whether perturbations use ℓ2 or ℓ∞ norms.Away from an optimum, adversarial training also controls a first-order gradient term in the Taylor expansion.

5. Experiments

Experiments evaluate search stability, generalizability, computational cost, anytime performance, and Hessian-norm trajectories across NAS-Bench-1Shot1 search spaces. SDARTS methods outperform baselines while reducing the Hessian norm during search.

  • 5. Experiments: Experiments cover NAS-Bench-1Shot1, CNN cells on CIFAR-10 and ImageNet, RNN cells on PTB, and comparisons with other regularization techniques.NAS-Bench-1Shot1 contains three CIFAR-10-based search spaces with 6,240, 29,160, and 363,648 architectures.
  • 5.1. Experiments on NAS-Bench-1Shot1: SDARTS-RS requires approximately 8h for 100 epochs on one GTX 1080 Ti, matching DARTS, whereas SDARTS-ADV requires 16h.The additional SDARTS-RS time is for random sampling; SDARTS-ADV adds forward and backward propagation for its adversarial attack.
  • 5.1. Experiments on NAS-Bench-1Shot1: DARTS produces deteriorating architectures at large search epochs, while GDAS, NASP, and PC-DARTS each exhibit distinct stability or generalization weaknesses.GDAS remains stable but converges prematurely, NASP fluctuates after early epochs, and PC-DARTS degenerates on Space 2.
  • 5.1. Experiments on NAS-Bench-1Shot1: SDARTS-RS outperforms all five baselines on all three NAS-Bench-1Shot1 search spaces.SDARTS-ADV performs even better, and its anytime test error continues decreasing beyond 80 search epochs.
  • 5.1. Experiments on NAS-Bench-1Shot1: The Hessian spectral norm of validation loss is correlated with solution quality, and SDARTS-RS and SDARTS-ADV consistently reduce it during training.Baselines generally increase λA_max, with some exceeding ten times their initial values after 100 epochs.

5.2. Experiments on CIFAR-10

On CIFAR-10, SDARTS regularization improves DARTS and its variants, with adversarial regularization achieving the lowest reported test error.

  • 5.2. Experiments on CIFAR-10: The experiments search CNN cells in the DARTS CIFAR-10 space with seven possible operations.The macro architecture stacks convolution cells eight times, with seven nodes per cell.
  • 5.2. Experiments on CIFAR-10: Table 1 compares the proposed methods with state-of-the-art image classifiers on CIFAR-10.The table reports the CIFAR-10 comparison summarized by the experiment.
  • 5.2. Experiments on CIFAR-10: SDARTS-RS decreases DARTS test error from 3.00% to 2.67%, while SDARTS-ADV further decreases it to 2.61%.Both methods also improve PC-DARTS and P-DARTS and reduce search-result variance.

5.3. Experiments on ImageNet

On ImageNet, both SDARTS regularization methods substantially outperform DARTS and improve PC-DARTS and P-DARTS, demonstrating transfer to a large-scale task.

  • 5.3. Experiments on ImageNet: The ImageNet transfer experiment uses networks with 14 cells and 48 initial channels, trained for 250 epochs.Training uses SGD with an annealing learning rate initialized at 0.5, momentum 0.9, and weight decay 3 × 10^-5.
  • 5.3. Experiments on ImageNet: The best run achieves 24.2% top-1 test error and 7.2% top-5 test error on ImageNet.This result ranks among popular NAS methods.
  • 5.3. Experiments on ImageNet: Both SDARTS-RS and SDARTS-ADV outperform DARTS by a large margin on ImageNet.Applying either regularization method to PC-DARTS and P-DARTS also improves accuracy.

5.4. Experiments on PTB

On PTB, SDARTS methods achieve strong language-model results in the RNN search space, while broader regularization comparisons show SDARTS-ADV leading most benchmarks.

  • 5.4. Experiments on PTB: The PTB search space contains five candidate functions and a single 12-node RNN cell.The candidate functions are tanh, relu, sigmoid, identity, and zero.
  • 5.4. Experiments on PTB: SDARTS-RS achieves validation/test perplexity of 58.7/56.4, while SDARTS-ADV achieves 58.3/56.1 on PTB.The methods outperform NAS methods with similar model size; LSTM + SE performs better using a handcrafted ensemble structure.
  • 5.4. Experiments on PTB: Table 3 compares state-of-the-art language models on PTB, where lower perplexity is better.The table provides the language-model comparison context for the reported PTB results.
  • 5.5. Comparison with Other Regularization: The comparison spans four simplified search spaces across CIFAR-10, CIFAR-100, and SVHN, with each method run four independent times.The final architecture is selected using validation accuracy.
  • 5.5. Comparison with Other Regularization: SDARTS-ADV is best on 11 of 12 benchmarks and takes both first and second places on 9 benchmarks.Its average improvements over DARTS, R-DARTS(L2), DARTS-ES, R-DARTS(DP), and PC-DARTS are 31.1%, 11.5%, 11.4%, 10.9%, and 5.3%, respectively.

5.6. Examine the Searched Architectures

The searched architectures reveal that DARTS favors parameter-free and wide, shallow structures, while SDARTS controls these biases across the simplified search spaces examined.

  • Parameter-free operations: Parameter-free operations such as skip connections can dominate DARTS-generated architectures, reducing representation capability and test accuracy.The paper reports this phenomenon across four simplified search spaces.
  • Parameter-free operations: SDARTS-RS and SDARTS-ADV control the proportion of parameter-free operations across all four search spaces.DARTS-ES is effective on S2 and S4 but fails on S3, where all discovered operations are skip connections.
  • Cell structure: DARTS favors wide and shallow cells because they often have smoother loss landscapes and faster convergence, although they may not generalize better.This motivates examining whether the searched architectures exhibit the same structural bias.
  • Cell structure: The best SDARTS-generated cell on CNN standard space has width 3c and depth 4 under the paper’s cell-width and depth definitions.Width sums the widths of intermediate nodes connected directly to inputs, while depth counts connections on the longest input-to-output path.

6. Conclusion

The paper concludes that SmoothDARTS stabilizes differentiable architecture search through perturbation-based regularization, with smoother landscapes and theoretical Hessian-norm regularization supported by experiments.

  • Conclusion: SmoothDARTS uses random smoothing or adversarial attack to improve the stability and generalizability of differentiable architecture search.The method is presented as a perturbation-based regularization.
  • Conclusion: SDARTS has a smoother loss landscape and a theoretical guarantee that it regularizes the Hessian norm of the validation loss.The paper reports extensive experiments showing effectiveness and performance beyond various regularization techniques.

7. Appendix

The appendix describes related differentiable and one-shot NAS methods, the experimental search setup and search spaces, and the definitions used to characterize cell structure.

  • Related work: ENAS trains shared parameters in a one-shot network, uses sampled-subnetwork validation error to train an RNN controller, and selects the highest-validation-accuracy architecture.The controller is updated with the REINFORCE rule.
  • Related work: DARTS continuously relaxes the architecture space with operation weights α, alternately optimizing network weights and architecture weights on training and validation sets.For evaluation, it retains the operation with the largest α on each edge.
  • Related work: GDAS and SNAS use Gumbel-Softmax to activate one operation per edge while retaining differentiability, reducing search memory cost.The passage specifically describes this shared technique for both methods.
  • Related work: NASP applies a proximal operation to compute a discrete architecture weight each epoch, then uses its gradient to update the continuous architecture weight.This is described as a modification of DARTS.
  • Related work: PC-DARTS evaluates a random proportion of channels to accelerate search and regularize DARTS’s bias toward parameter-free operations.The regularization role is attributed to the method’s author.
  • Experimental setup: The experiments train the mixture architecture for 50 epochs on a 50K CIFAR-10 split, using SGD for network weights and Adam for architecture weights.The network optimizer uses momentum 0.9 and weight decay 3 × 10^-4; the architecture optimizer uses learning rate 3e-4 and weight decay 1e-3.
Loading 2002.05283v3…