PaperlayerView exact revision on arXiv

Source-linked AI summary

DARTS-: Robustly Stepping out of Performance Collapse Without Indicators

Xiangxiang Chu, Xiaoxing Wang, Bo Zhang, Shun Lu, Xiaolin Wei, Junchi Yan

arXiv:2009.01027v2cs.LGcs.AIcs.CVstat.ML

TL;DR

DARTS suffers from performance instability and collapse, while existing robustification methods rely on indicators that can reject good architectures. DARTS- factors out skip connections’ gradient advantage with an auxiliary skip connection, producing fairer competition and substantially improved robustness across settings.

  • Problem

    DARTS performance collapse limits its application, while indicator-based robustification can reject good architectures and depends on noisy search signals.

  • Method

    DARTS- adds an auxiliary skip connection between nodes to take over skip connections’ gradient advantage and create fairer competition among candidate operations.

  • Results

    DARTS- robustly obtains state-of-the-art results on 4 search space with 3× fewer search cost than R-DARTS, and improves P-DARTS and PC-DARTS accuracy on CIFAR-10.

  • Takeaways & Limitations

    The method stabilizes search without relying on handcrafted regularization tricks and can combine with other DARTS variants.

  • Takeaways & Limitations

    Hessian eigenvalue indicators can reject good models, so they are not ideal as universal collapse signals.

Abstract

from arXiv · show

Despite the fast development of differentiable architecture search (DARTS), it suffers from long-standing performance instability, which extremely limits its application. Existing robustifying methods draw clues from the resulting deteriorated behavior instead of finding out its causing factor. Various indicators such as Hessian eigenvalues are proposed as a signal to stop searching before the performance collapses. However, these indicator-based methods tend to easily reject good architectures if the thresholds are inappropriately set, let alone the searching is intrinsically noisy. In this paper, we undertake a more subtle and direct approach to resolve the collapse. We first demonstrate that skip connections have a clear advantage over other candidate operations, where it can easily recover from a disadvantageous state and become dominant. We conjecture that this privilege is causing degenerated performance. Therefore, we propose to factor out this benefit with an auxiliary skip connection, ensuring a fairer competition for all operations. We call this approach DARTS-. Extensive experiments on various datasets verify that it can substantially improve robustness. Our code is available at https://github.com/Meituan-AutoML/DARTS- .

1 INTRODUCTION

DARTS- proposes an indicator-free stabilization strategy that removes skip connections’ unfair advantage during search. Experiments report robust performance across search spaces and datasets, including improved combinations with DARTS variants.

  • Motivation: DARTS performance collapse has been linked to superfluous skip connections, motivating indicators such as Hessian eigenvalues for detecting collapse.These approaches use observed deterioration signals rather than directly addressing its cause.
  • Method: DARTS- introduces an auxiliary skip connection to remove the unfair advantage of candidate skip operations during the search phase.The auxiliary connection is shown schematically between every two nodes.
  • Method: The approach targets robustness without handcrafted indicators or restrictions that interfere with searching or discretization.It also addresses the question of achieving robustness without tuning extra hyper-parameters.
  • Results: Across seven search spaces and three datasets, DARTS- robustly obtains state-of-the-art results on 4 search space with 3× fewer search cost than R-DARTS.R-DARTS requires four independent runs to report final performance.
  • Results: DARTS- improves P-DARTS accuracy by 0.8% and PC-DARTS accuracy by 0.25% on CIFAR-10.The method is presented as a seamless plug-in combination with these DARTS variants.

2 RELATED WORK

Related work addresses DARTS instability through architecture-search background and robustification strategies based on indicators, perturbations, or human-designed interventions.

  • Neural architecture search and DARTS variants: Differentiable Architecture Search uses weight-sharing and gradient descent to make neural architecture search efficient.The broader search task spans image classification, detection, segmentation, and machine translation.
  • Robustifying DARTS: Robustifying DARTS methods include Hessian-eigenvalue indicators, perturbation-based regularization, and variants that apply stronger human intervention.Indicator-based methods depend on how accurately the indicator correlates with performance collapse.
  • Robustifying DARTS: DARTS- is named as an inward approach that avoids designing new indicators, adding extra cost, and introducing new hyper-parameters.This contrasts with outward approaches that modify the search using such additions.

3 DARTS-

DARTS- separates skip connections’ training-stabilization role from their role as candidate operations by adding an auxiliary skip connection during search. This reduces skip connections’ optimization advantage and stabilizes architecture search without relying on handcrafted collapse indicators.

  • 3.1 MOTIVATION: Skip connections alleviate gradient vanishing because shallow-layer gradients include contributions from deeper layers.A trainable coefficient β controls gradient memory during back-propagation and stabilizes training.
  • 3.1 MOTIVATION: β converges toward 1 after 40 epochs from initializations 0, 0.5, and 1.0, showing residual training favors a large skip coefficient.The experiment uses ResNet50 trained and tested on CIFAR-10.
  • 3.1 MOTIVATION: In DARTS, βskip can increase and dominate architecture parameters because it both stabilizes supernet training and serves as a candidate operation.DARTS- addresses this two-fold role rather than relying on indicators that may reject good models.
  • 3.2 STEPPING OUT OF THE PERFORMANCE COLLAPSE: DARTS- adds a fixed auxiliary skip connection between every two cell nodes, factoring out skip connections’ training advantage and equalizing operation competition.The auxiliary connection stabilizes supernet training even when βskip is small.
  • 3.2 STEPPING OUT OF THE PERFORMANCE COLLAPSE: DARTS- decays β linearly to 0 during search, degenerating to standard DARTS at the end to avoid affecting discretization.Algorithm 1 updates network weights and architecture parameters iteratively using training and validation losses.
  • 3.2 STEPPING OUT OF THE PERFORMANCE COLLAPSE: With β ≫ βskip, the convergence factor λ becomes insensitive to βskip and depends more on βconv, reducing skip connections’ optimization privilege.The method uses β = 1.0 initially while βskip is commonly 0.15.

4 EXPERIMENTS

Experiments across search spaces and datasets show that DARTS- improves robustness, search efficiency, and compatibility with other DARTS variants. The method also retains or improves performance under altered decay settings and auxiliary-branch choices.

  • CIFAR-10 and CIFAR-100: On CIFAR-10, DARTS- averages 97.36% top-1 accuracy across six independent runs, with the best cell reaching 97.5%.The search cost is reported as 3× lower than R-DARTS.
  • ImageNet: On ImageNet, DARTS-A reaches 76.2% top-1 validation accuracy, versus 66.4% for directly applying DARTS in the same search space.With auto-augmentation and squeeze-and-excitation, it reaches 77.8%.
  • NAS-Bench-201: On NAS-Bench-201, DARTS- achieves a new state of the art, with its best result nearly reaching the optimum.The comparison averages four search runs.
  • Orthogonal combination with other variants: Combining DARTS- with P-DARTS improves CIFAR-10 top-1 accuracy by +0.8% after removing P-DARTS’s strong skip-connection and dropout priors.Without those priors, P-DARTS has 3.42% average test error and suffers severe collapse.
  • Orthogonal combination with other variants: Combining DARTS- with PC-DARTS improves CIFAR-10 top-1 accuracy by +0.26% on average and reduces sensitivity to the channel proportion K.The comparison uses K=2, described as a bad configuration in the original PC-DARTS paper, while retaining lower memory cost and runtime.

5 ANALYSIS AND DISCUSSIONS

DARTS- produces smoother, less perturbation-sensitive validation landscapes, yet its Hessian eigenvalues can grow substantially without indicating poor final performance. This conflicts with eigenvalue-based stopping criteria, which may reject good models.

  • 5 ANALYSIS AND DISCUSSIONS: DARTS- can develop nearly monotonically increasing maximal Hessian eigenvalues while still producing final models with state-of-the-art performance.The paper reports that these models could be mistakenly rejected or never visited under eigenvalue criteria.
  • 5 ANALYSIS AND DISCUSSIONS: Existing Hessian-eigenvalue indicators therefore risk rejecting good architectures when used to detect performance collapse.The authors identify this as a fatal drawback of indicator-based approaches.
  • 5 ANALYSIS AND DISCUSSIONS: DARTS- is less sensitive to normalized perturbations of architectural weights α than DARTS, with a more gently descending contour map.The reported derived-model accuracies are 94.84% for DARTS and 97.58% for DARTS-.

6 CONCLUSION

DARTS- stabilizes differentiable architecture search by giving the auxiliary skip branch the gradient-advantage role, allowing candidate operations to compete more fairly. The authors report robust performance across search spaces and datasets, with lower search cost than RobustDARTS.

  • 6 CONCLUSION: DARTS- uses an auxiliary skip connection to take over the gradient advantage of the candidate skip operation.This factors out the residual block’s advantageous contribution during search.
  • 6 CONCLUSION: The resulting fair competition helps the bi-level optimization distinguish good operations from bad ones, making collapse seldom occur across search spaces and datasets.The paper presents this as the mechanism underlying the method’s more stable search process.
  • 6 CONCLUSION: DARTS- steadily outperforms RobustDARTS under strictly controlled settings with 3× fewer search cost.The method can operate stand-alone or cooperate with orthogonal DARTS improvements.
  • 6 CONCLUSION: The paper argues that Hessian-eigenvalue indicators can reject good models and that handcrafted regularization may be more critical than proposed indicators for finding good models.It frames controlling the search process without imposing limitations or priors on the final model as a promising direction.

A.2.1 TRAINING DETAILS

The experiments report reduced-search-space results on CIFAR-10 and CIFAR-100, alongside eigenvalues, and describe separate ImageNet and COCO training settings. The reported comparisons follow established experimental protocols and dataset-specific configurations.

  • A.2.1 TRAINING DETAILS: Table 10 reports CIFAR-100 results in search space S0, while Table 9 covers reduced spaces S1-S4.The appendix distinguishes the CIFAR-100 S0 results from the reduced-space comparison.
  • A.2.1 TRAINING DETAILS: Table 9 compares CNN architectures in reduced search spaces S1-S4 on CIFAR-10 and CIFAR-100 using mean±std test error and corresponding eigenvalues.Three architectures retrained from scratch are used for the reported averages, following the settings of Zela et al. (2020).
  • A.2.1 TRAINING DETAILS: ImageNet training follows the MnasNet setting and includes squeeze-and-excitation for comparison with EfficientNet.Large-model distillation and MobileNetV3’s efficient head are excluded for fairness.
  • A.2.1 TRAINING DETAILS: COCO models are trained and evaluated for 12 epochs with batch size 16, using learning-rate reductions at epochs 8 and 11.The initial learning rate is 0.01 and is reduced by 0.1 at both milestones.

A.2.2 FURTHER DISCUSSIONS ON FAILURE OF EIGENVALUE

Additional experiments show that DARTS- models can improve throughout search even as Hessian eigenvalues change rapidly. This provides a counterexample to eigenvalue-based early stopping and supports continuing the search under DARTS-.

  • A.2.2 FURTHER DISCUSSIONS ON FAILURE OF EIGENVALUE: 96.5% to 97.4%: the accuracy of inferred models continues growing across models sampled every 10 epochs during DARTS- search.Each sampled model is trained from scratch under the same settings.
  • A.2.2 FURTHER DISCUSSIONS ON FAILURE OF EIGENVALUE: Eigenvalue-based early stopping would fail in this setting because later sampled models achieve higher accuracy despite changing eigenvalues.The experiment is presented as a counterexample to RobustDARTS.
  • A.2.2 FURTHER DISCUSSIONS ON FAILURE OF EIGENVALUE: The authors attribute this behavior to the auxiliary skip branch regularizing supernet overfitting while leaving architectural weights to represent candidate-operation ability.They use this observation to question the need for a costly eigenvalue indicator.
  • A.2.2 FURTHER DISCUSSIONS ON FAILURE OF EIGENVALUE: Among five sampled models in the S0 CIFAR-10 example, the model associated with the highest eigenvalue has the best performance.Figure 4 describes the models as sampled during the DARTS- search process.

A.2.3 MORE ABLATION STUDIES

Additional ablations examine sensitivity to β initialization and compare architectural-parameter loss landscapes for DARTS and DARTS-. These visualizations indicate more inflated DARTS- slopes and suggest better convergence of the over-parameterized network.

  • Figure 6 supplements sensitivity analysis with training-loss curves for different initial β values.
  • Figures 7 and 8 visualize validation-accuracy landscapes and contour maps against architectural weights α for DARTS and DARTS- in S0 and S3.
  • DARTS- has more inflated loss-landscape slopes, suggesting better convergence of the over-parameterized network.

A.4 LIST OF EXPERIMENTS

The paper’s experiments are catalogued in Table 12 and span architecture visualizations, search spaces, datasets, baseline variants, and βskip schedules. The listed figures include comparisons of DARTS- with P-DARTS and PC-DARTS, as well as CIFAR-10 and CIFAR-100 results.

  • Table 12 summarizes the experiments conducted in the paper and links them to the corresponding figures and tables.
  • Figures 9 and 11 show the best normal and reduction cells found across search spaces S0-S4 on CIFAR-10 and CIFAR-100.
  • Figure 10 presents the architecture of DARTS-A searched on ImageNet.
  • Figures 12–15 compare cells found by P-DARTS and PC-DARTS with or without the proposed auxiliary skip connections in the standard DARTS search space on CIFAR-10.
  • Figures 16–21 examine keeping βskip fixed at 1 or decaying it during the final 50 search epochs across the DARTS search space and spaces S2-S3 on CIFAR-10.
Loading 2009.01027v2…