PaperlayerView exact revision on arXiv

Source-linked AI summary

DARTS+: Improved Differentiable Architecture Search with Early Stopping

Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, Zhenguo Li

arXiv:1909.06035v2cs.CVcs.LG

TL;DR

DARTS can collapse after prolonged searching, with overfitting in its search phase linked to a growing number of skip-connects. DARTS+ addresses this issue by explicitly early-stopping the search at adaptive criteria, achieving strong results across benchmark datasets and search spaces.

  • Problem

    DARTS suffers performance collapse after many search epochs, as selected architectures accumulate skip-connects.

  • Method

    DARTS+ adds an adaptive early-stopping criterion to halt the search when DARTS starts to collapse.

  • Results

    DARTS+ achieves state-of-the-art results across benchmark datasets and search spaces, including 2.32% test error on CIFAR10 and 14.87% on CIFAR100.

  • Takeaways & Limitations

    Early stopping is presented as an efficient way to avoid DARTS collapse while reducing search costs, and can also be applied to various search spaces and DARTS progress variants.

  • Takeaways & Limitations

    Adjusting learning rates, data proportions, or skip-connect regularization only alleviates collapse at certain epochs; collapse eventually reappears.

Abstract

from arXiv · show

Recently, there has been a growing interest in automating the process of neural architecture design, and the Differentiable Architecture Search (DARTS) method makes the process available within a few GPU days. However, the performance of DARTS is often observed to collapse when the number of search epochs becomes large. Meanwhile, lots of "{\em skip-connect}s" are found in the selected architectures. In this paper, we claim that the cause of the collapse is that there exists overfitting in the optimization of DARTS. Therefore, we propose a simple and effective algorithm, named "DARTS+", to avoid the collapse and improve the original DARTS, by "early stopping" the search procedure when meeting a certain criterion. We also conduct comprehensive experiments on benchmark datasets and different search spaces and show the effectiveness of our DARTS+ algorithm, and DARTS+ achieves $2.32\%$ test error on CIFAR10, $14.87\%$ on CIFAR100, and $23.7\%$ on ImageNet. We further remark that the idea of "early stopping" is implicitly included in some existing DARTS variants by manually setting a small number of search epochs, while we give an {\em explicit} criterion for "early stopping".

1 Introduction

DARTS enables fast neural architecture search but can collapse after prolonged searching as skip-connects accumulate. DARTS+ attributes this failure to search-phase overfitting and introduces explicit early stopping, validated across datasets and search spaces.

  • DARTS performs fast architecture search by encoding the search space with continuous parameters and optimizing a one-shot model with gradient-based bi-level optimization.
  • Performance collapse occurs after certain search epochs, when selected architectures contain dramatically more skip-connects and perform poorly.
  • DARTS+ adds an explicit early-stopping paradigm that halts searching at a criterion intended to avoid collapse.
  • 2.32% test error on CIFAR10 and 14.87% test error on CIFAR100 were achieved in the DARTS search space with less than 0.4 GPU days.
  • 23.7% top-1 error was achieved on ImageNet after transfer, while introducing an SE-Module produced 22.5% top-1 error.
  • The authors report extensive experiments across benchmark datasets and search spaces, with state-of-the-art results on all evaluated settings.

2 Collapse of DARTS

DARTS collapse occurs when prolonged search selects increasingly many skip-connects, producing shallow, poorly performing architectures. The paper attributes this behavior to overfitting in the one-shot model, which creates a training–validation gap and favors skip-connects, especially in later cells.

  • Collapse issue: DARTS collapse is the performance drop that occurs after sufficiently many search epochs, when selected architectures contain dramatically more skip-connects.The resulting architectures are reported to perform poorly.
  • DARTS formulation: DARTS represents each cell as a directed acyclic graph whose edges mix candidate operations using architecture parameters.Candidate operations include zero, skip-connect, convolution, and max-pooling; the resulting hyper-network is the one-shot model.
  • DARTS formulation: The final discrete architecture selects the highest-probability non-zero operation on each edge and two incoming edges for every intermediate node.These choices are derived from the architecture parameters after search.
  • Collapse issue: Many skip-connects make the selected network shallow, reducing its learnable parameters and expressive power relative to deeper architectures.On CIFAR100, skip-connect architecture weights grow with search epochs, and later cells become especially shallow.
  • Overfitting and analysis: The paper links collapse to overfitting of model weights: training error becomes much lower than validation error as the over-parameterized one-shot model fits training data.The authors describe this as a large training–validation error gap during bi-level optimization.
  • Overfitting and analysis: When cells share one architecture, later-cell preference for skip-connects is propagated toward earlier cells, causing the selected architecture to degrade.With distinct cell architectures, deeper learnable operations tend to appear in early cells while skip-connects are preferred in later cells.

3 The Early Stopping Methodology

DARTS+ avoids collapse by stopping the search adaptively when architecture choices saturate, using simple criteria based on skip-connect counts or stable architecture-parameter rankings. The method is designed to improve performance while reducing search cost and can complement other DARTS-based techniques.

  • Early stopping rationale: DARTS+ early stops the search when DARTS begins to collapse, addressing overfitting while reducing search costs.The method is based on DARTS and uses an adaptive stopping criterion.
  • Architecture selection: The ranking of architecture parameters matters because only the operation with the maximum α is selected, making unstable rankings too noisy for architecture selection.When the ranking becomes stable, the corresponding learnable operations are suitable for selection.
  • Criterion 1: Criterion 1 stops searching when a normal cell contains two or more skip-connects.Its simplicity is motivated by the observation that too many skip-connects hurt performance, whereas an appropriate number can help information transfer and stabilize training.
  • Relation to prior methods: DARTS+ differs from P-DARTS by stopping during search rather than replacing redundant skip-connects only after searching finishes.P-DARTS also uses manually shortened searches and other interventions, while DARTS+ applies explicit stopping criteria.
  • Criterion 2: Criterion 2 stops when the ranking of architecture parameters α for learnable operations remains stable for a determined number of epochs, such as 10.Stable rankings indicate that the search procedure has reached saturation; experiments suggest the ten-epoch setting is flexible because rankings stable for more than six epochs can suffice.

4 Experiments and Analysis

Experiments across benchmark datasets and search spaces show that explicit early stopping prevents DARTS collapse and yields strong architectures with limited search time. DARTS+ performs well in the DARTS, MobileNetV2, and ResNet-related settings, including CIFAR10, CIFAR100, Tiny-ImageNet-200, and ImageNet.

  • Experimental Setup: Experiments evaluate DARTS+ across CIFAR10, CIFAR100, Tiny-ImageNet-200, and ImageNet using architecture search and evaluation stages.The study also considers DARTS, MobileNetV2, and ResNet search spaces.
  • Search Results and Analysis: Early stopping produces good architectures at both criteria, whereas the original DARTS performs worse when search epochs increase.The two stopping criteria achieve comparable performance because their stopping points are close.
  • Search Results and Analysis: DARTS+ searches efficiently, stopping at about epoch 35 on CIFAR10, epoch 18 on CIFAR100, and epoch 10 on Tiny-ImageNet-200.The corresponding search times are 0.4 GPU days, 0.2 GPU days, and an unstated duration; ImageNet requires 6.8 GPU days for 200 epochs.
  • Architecture Evaluation on CIFAR10 and CIFAR100: DARTS+ achieves 2.32% test error on CIFAR10 and 14.87% test error on CIFAR100 with the simple early-stopping paradigm.The paper reports these as the best results in the summarized comparison and describes DARTS+ as simpler than modified DARTS algorithms.
  • Architecture Evaluation on Tiny-ImageNet-200: DARTS+ achieves 28.3% and 27.6% test error on Tiny-ImageNet-200 with Criteria 1 and 2, respectively.The architecture searched with DARTS performs much worse because it suffers from collapse and contains many skip-connects.
  • Architecture Evaluation on ImageNet: On ImageNet, DARTS+ reaches 23.9%/7.4% top-1/top-5 error when searched directly and 23.7%/7.2% after transfer from CIFAR100.With an SE-module and additional augmentation, the transferred architecture reaches 22.5%/6.4% top-1/top-5 error with 3M additional FLOPs.

5 Conclusion

The paper attributes DARTS collapse mainly to one-shot-model overfitting and introduces early stopping in DARTS+ to avoid it. Experiments report successful searches across benchmark datasets, including ImageNet, with state-of-the-art performance.

  • Conclusion: DARTS+ identifies overfitting of the one-shot model as the main cause of DARTS collapse.The collapse is associated with performance degradation after extended search.
  • Conclusion: DARTS+ introduces early stopping to avoid collapse, with criteria applicable to various search spaces and potentially to recent DARTS progress.The conclusion describes the paradigm as enabling successful searches on benchmark datasets with limited GPU days.
  • Conclusion: Experiments report state-of-the-art performance from architectures searched on benchmark datasets including large-scale ImageNet.The conclusion summarizes the empirical evidence as covering multiple benchmark settings.

A Proof of Lemma 1

The proof analyzes gradient dynamics in a normalized setting and shows that sufficiently large variance drives the identity-mixture coefficient upward. It also establishes that the threshold σ0(r) decreases with r and is not sensitive to α0.

  • The proof defines Wα = α0I + α1W with α1 = 1 − α0 and analyzes gradients of W and wr under the training loss.
  • When training variance is small and ∥v∥ is limited, the gradient of wr is expected to align with Wαe, causing wr to become parallel to e.
  • The corresponding gradient of W aligns with λ1wre⊤, so gradient descent drives W toward a matrix proportional to ee⊤.
  • Normalization of W*x yields a specific scale η for W* = ηee⊤, completing the proof of the first proposition.
  • There exists σ0(r) ∈ (0, 1) such that g(r, σv) < 0 for σv > σ0(r), which makes α0 increase under gradient descent.
  • The threshold σ0(r) decreases as r increases and is not sensitive to α0; Figure 5 reports its numerical value for α0 = 0.5.

B.1 Architecture Search

The experiments search DARTS, MobileNetV2, and ResNet-based spaces using alternating optimization and explicit early-stopping criteria. The selected cells generally contain mostly convolutions and few skip-connects.

  • DARTS Search Space: The DARTS search space contains eight candidate operations, including skip-connect, pooling, separable and dilated convolutions, and zero.
  • DARTS Search Space: CIFAR searches use an eight-cell one-shot model, split training data for weights and architecture parameters, and search for at most 60 epochs.
  • DARTS Search Space: Tiny-ImageNet uses the CIFAR configuration with an added stride-2 convolution, while retaining the early-stopping criterion.
  • DARTS Search Space: Across datasets, one-shot weights and architecture parameters are optimized alternately, and architecture parameters determine the final cell structure.
  • DARTS Search Space: The cells searched by DARTS+ contain most convolutions and a few skip-connects.
  • MobileNetV2 and ResNet Search Spaces: DARTS+ architectures are also selected with early-stopping Criterion 2 in MobileNetV2 and ResNet-based search spaces.

B.2 Architecture Evaluation

The selected architectures are evaluated with established configurations across CIFAR, Tiny-ImageNet, ImageNet, MobileNetV2, and ResNet search spaces. DARTS+ reaches 72.43% and 81.23% in the MobileNetV2 and ResNet spaces, respectively.

  • DARTS Search Space: Evaluation follows prior-work configurations and hyperparameters, with larger networks and extended training used for CIFAR and Tiny-ImageNet.
  • DARTS Search Space: ImageNet evaluation uses 14 cells, 48 initial channels, 800 epochs, batch size 2048, and eight Nvidia Tesla V100 GPUs.
  • DARTS Search Space: Architectures searched directly on ImageNet and transferred from CIFAR100 are both evaluated to test transferability.
  • MobileNetV2 and ResNet Search Spaces: 72.43% is achieved in the MobileNetV2 search space and 81.23% in the ResNet search space.

C.1 More Illustrations on the Collapse of DARTS

Additional evaluations on CIFAR10, CIFAR100, and Tiny-ImageNet further examine DARTS collapse and the effectiveness of the proposed early-stopping criteria.

  • Experiments evaluate cells selected at different epochs on CIFAR10, CIFAR100, and Tiny-ImageNet-200.
  • The results support conclusions about DARTS collapse and validate both early-stopping criteria.

C.2 Implicit Early Stopping in PC-DARTS

Experiments with PC-DARTS on CIFAR100 indicate that longer searches select increasingly many skip-connects, suggesting possible collapse. The 50-epoch setting used by DARTS and PC-DARTS therefore functions as an implicit early-stopping scheme.

  • PC-DARTS was evaluated on CIFAR100 for 600 search epochs using its original code and settings.Selected normal cells at 200, 400, and 600 epochs are shown in Fig. 12.
  • Additional results are presented for CIFAR10, CIFAR100, and Tiny-ImageNet-200 using the same figure interpretation as Fig. 2.
  • The selected normal cells contained 2 skip-connects at 200 epochs and 5 at 600 epochs.The increase in skip-connects implies that PC-DARTS may suffer from collapse with more search epochs.
  • Training for just 50 epochs, as used in DARTS and PC-DARTS, is an implicit early-stopping scheme for obtaining better architectures.
Loading 1909.06035v2…