PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding and Robustifying Differentiable Architecture Search

Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, Frank Hutter

arXiv:1909.09656v2cs.LGcs.AIcs.CVstat.ML

TL;DR

DARTS can select degenerate architectures that generalize poorly even while validation loss improves. This paper studies the role of architecture-space curvature and tests early stopping and inner-objective regularization, finding more robust behavior across search spaces and tasks.

  • Problem

    DARTS is efficient but has been reported to fail on new problems, sometimes performing no better than random search.

  • Method

    The paper analyzes the Hessian spectrum of validation loss with respect to architecture parameters and robustifies DARTS using early stopping, data augmentation, and L2 regularization of the inner objective.

  • Results

    The proposed robustifications overcome standard DARTS failure modes across 12 NAS benchmarks, with findings also holding for language modelling and disparity estimation.

  • Takeaways & Limitations

    Tracking and controlling architecture-space curvature provides simple practical variations of DARTS that are substantially more robust.

  • Takeaways & Limitations

    The search space S4 was specifically constructed to showcase DARTS selecting the suboptimal Noise operator.

Abstract

from arXiv · show

Differentiable Architecture Search (DARTS) has attracted a lot of attention due to its simplicity and small search costs achieved by a continuous relaxation and an approximation of the resulting bi-level optimization problem. However, DARTS does not work robustly for new problems: we identify a wide range of search spaces for which DARTS yields degenerate architectures with very poor test performance. We study this failure mode and show that, while DARTS successfully minimizes validation loss, the found solutions generalize poorly when they coincide with high validation loss curvature in the architecture space. We show that by adding one of various types of regularization we can robustify DARTS to find solutions with less curvature and better generalization properties. Based on these observations, we propose several simple variations of DARTS that perform substantially more robustly in practice. Our observations are robust across five search spaces on three image classification tasks and also hold for the very different domains of disparity estimation (a dense regression task) and language modelling.

1 INTRODUCTION

DARTS greatly reduces neural architecture search costs, but its reported success is not robust across new problems. This paper investigates these failures and develops more robust variants.

  • DARTS reduces neural architecture search costs to the same order of magnitude as training one neural network.This efficiency expanded NAS to more expensive problems, including semantic segmentation.
  • Researchers have reported that DARTS sometimes performs no better than random search.
  • The paper identifies 12 benchmarks where standard DARTS produces degenerate architectures with poor test performance.
  • The authors connect architecture generalization error with the dominant Hessian eigenvalue of validation loss and propose early stopping and stronger inner-objective regularization.
  • The findings hold across image-recognition benchmarks, language modelling, and disparity estimation, supporting substantially more robust DARTS versions.

2 BACKGROUND AND RELATED WORK

NAS automatically designs neural networks, while DARTS makes the search differentiable through continuous architecture parameters and alternating optimization of weights and architecture.

  • 2.3 NEURAL ARCHITECTURE SEARCH: NAS automatically designs neural network architectures instead of relying on manual trial and error.
  • 2.3 NEURAL ARCHITECTURE SEARCH: Earlier NAS methods commonly used reinforcement learning or evolutionary algorithms, but their computational expense restricted applicability to simpler tasks and smaller datasets.
  • 2.4 DIFFERENTIABLE ARCHITECTURE SEARCH (DARTS): DARTS relaxes the discrete search problem into a continuous one by using weighted sums of candidate operations, whose real-valued weights parameterize the architecture.
  • 2.4 DIFFERENTIABLE ARCHITECTURE SEARCH (DARTS): DARTS alternates optimization of network weights on training data and architecture parameters on validation data within a bi-level formulation.
  • 2.4 DIFFERENTIABLE ARCHITECTURE SEARCH (DARTS): The search approximates the lower-level optimization using a single gradient step, then prunes operations to form a discrete cell.

3 WHEN DARTS FAILS

Across several search spaces and tasks, standard DARTS often selects degenerate architectures despite smoothly improving validation loss. The failures include parameter-free or harmful operations and late-stage overfitting.

  • Search spaces: The study evaluates four simplified DARTS-like search spaces across CIFAR-10, CIFAR-100, and SVHN, plus additional spaces for disparity estimation and known-optimum benchmarking.
  • Search spaces: S1 is a small pre-optimized space created by iteratively removing the least-important operations from the original DARTS space.
  • Search spaces: S2 uses 3 × 3 SepConv and SkipConnect, while S3 adds Zero and S4 pairs 3 × 3 SepConv with Noise.
  • Observed failures: Standard DARTS selects mostly parameter-less skip connections in S1-S3 and selects the harmful Noise operation five times out of eight in S4.
  • Observed failures: In S5, DARTS approaches the global optimum but later suffers deteriorating test performance while validation error continues converging, indicating architectural overfitting.
  • Observed failures: Across the tested spaces, validation error can converge even as selected architectures degenerate and generalize poorly.

4 THE ROLE OF DOMINANT EIGENVALUES OF ∇2αLvalid

DARTS can reduce validation error while producing architectures whose test error rises with the dominant eigenvalue of the validation-loss Hessian. This curvature also predicts discretization-related accuracy drops, motivating early stopping and regularization to control eigenvalues.

  • DARTS validation error converges even when the selected cell structures are poor.
  • The dominant eigenvalue increases during standard DARTS search while validation error decreases and test error rises.The eigenvalue serves as a proxy for sharpness.
  • 0.867 Pearson correlation links the dominant architectural Hessian eigenvalue to test error on S1 CIFAR-10.The comparison used 24 architectures from standard and regularized DARTS variants.
  • Low curvature never produced large performance drops after discretizing the continuous architecture into a discrete one.The paper measured the drop between the continuous α* and discretized αdisc and observed a similar relationship using accuracy drop.
  • Early stopping monitors the dominant eigenvalue rather than validation performance, which keeps improving throughout architecture search.The heuristic smooths the eigenvalue over k = 5 epochs and stops when it increases too much.
  • Early stopping avoids exploding eigenvalues, shortens search, and improved DARTS across all tested image-classification settings without harming it.The mechanism did not use test data.

5 REGULARIZATION OF INNER OBJECTIVE

Regularizing the inner objective during architecture search reduces curvature and improves DARTS robustness across image, disparity-estimation, and language-modelling settings. The paper proposes adaptive and multi-run robustifications that avoid manual regularization tuning.

  • Regularization strategy: Search-phase regularization changes the inner landscape so architectural optimization is redirected toward flatter validation-loss regions.The paper regularizes only the search phase, not final training and evaluation.
  • Data augmentation: Data augmentation with Cutout and ScheduledDropPath improved DARTS and DARTS-ES test performance in all evaluated image-classification cases while keeping the dominant eigenvalue relatively low.These methods were tested across three image-classification datasets and search spaces S1-S4.
  • Data augmentation: Data augmentation avoided degenerate architectures and substantially improved disparity-estimation results on S6.The study varied standard augmentations such as shearing and brightness change and evaluated average end point error.
  • L2 regularization: Higher L2 factors 3^i · 10^-4 for i ∈ {1, 3, 9, 27, 81} significantly improved DARTS test performance across datasets and spaces while keeping the dominant eigenvalue low.The default DARTS setting corresponds to i = 1; DARTS with early stopping also benefited from additional regularization.
  • Practical robustification: DARTS-ADA consistently improved over standard DARTS, while RobustDARTS yielded the best performance and was recommended for practice because it is easier to implement.RobustDARTS selects among four searches using different regularization amounts; DARTS-ADA increases regularization adaptively.

6 CONCLUSIONS

The paper relates DARTS generalization to validation-loss Hessian eigenvalues and develops more robust search procedures through early stopping and regularization. It also describes DARTS as a bi-level optimization whose exact architectural gradient is approximated for computational efficiency.

  • 6 CONCLUSIONS: Standard DARTS can produce degenerate architectures with large Hessian eigenvalues and poor generalization.The dominant validation-loss Hessian eigenvalue is associated with architecture generalization error.
  • 6 CONCLUSIONS: Tracking the largest Hessian eigenvalue provides a basis for an early stopping criterion that makes DARTS substantially more robust.The proposed criterion stops search based on the largest eigenvalue.
  • 6 CONCLUSIONS: Regularizing the inner objective helps control Hessian eigenvalues and improves the generalization of discovered architectures.The conclusion reports consistency across image recognition, language modelling, and disparity estimation.
  • DARTS optimization: DARTS formulates architecture search as bi-level optimization, with validation loss as the outer objective and training loss as the inner objective.Both losses depend on architecture parameters α and network weights w.
  • DARTS optimization: DARTS approximates the expensive optimum w∗(α) using a single gradient-descent step before computing the architectural gradient.Exact optimization of w to convergence after every α update would be computationally expensive.
  • DARTS optimization: Finite differences reduce the cost of the vector-matrix product used in DARTS’s architectural-gradient computation.The approximation evaluates gradients around perturbed weights w+ and w−.

C FINAL ARCHITECTURE EVALUATION

Final architecture evaluation retrains scaled-up architectures from scratch under specified dataset- and search-setting configurations. The evaluation keeps the final-training regularization settings fixed to the original DARTS values.

  • Final retraining: Architectures found during search are scaled up and retrained from scratch to obtain final test performance.For CIFAR-100 and SVHN, the evaluation uses 16 initial filters and 8 cells.
  • CIFAR-10 settings: CIFAR-10 evaluations use 36 initial filters and 20 cells for ScheduledDropPath experiments, but reduce filters to 16 for S2 and S4 to avoid memory issues.The reduction applies because more-regularized cells usually contain only separable convolutions.
  • Final training regularization: Final architecture training uses ScheduledDropPath increasing to 0.2, Cutout probability 1.0, and L2 regularization factor 3 · 10^-4.These values remain fixed even when regularization factors are varied during search.

D ADDITIONAL EMPIRICAL RESULTS

Additional experiments examine how L2 regularization, data augmentation, and drop-path affect validation loss, Hessian spectra, discretization, and architecture performance. Stronger regularization generally reduces curvature and is associated with better generalization, while excessive L2 can underfit.

  • L2 regularization: Increasing L2 regularization initially reduces architectural-parameter overfitting and can find the global minimum, but excessive L2 causes underfitting and higher validation and test error.The figure specifically reports this behavior on S5 and CIFAR-10.
  • DARTS-ADA: DARTS-ADA is evaluated with R = 3 · 10^-4, Rmax = 3 · 10^-2, and η = 10 across the image-classification search spaces and datasets.Its procedure can stop and roll back to a selected epoch before continuing with increased regularization.
  • Additional evaluations: The experiments compare validation and test errors, eigenvalue trajectories, full Hessian spectra, discretization accuracy drops, and found-architecture performance across search settings.The cited figures vary ScheduledDropPath and L2 factors and include comparisons involving Cutout and RandomNAS with weight sharing.
  • Hessian curvature: Increasing inner-objective regularization through L2 or data augmentation keeps the largest Hessian eigenvalue small during search.The paper uses this observation to explain improved architecture generalization.
  • Hessian spectrum: Stronger regularization reduces not only the dominant Hessian eigenvalue but also the absolute magnitudes of other eigenvalues.This indicates a sharper objective landscape toward many directions when regularization is weaker.

E.1 DATASETS

The disparity-estimation experiments train AutoDispNet on FlyingThings3D and test on both FlyingThings3D and Sintel using synthetic stereo data with ground-truth disparity maps.

  • FlyingThings3D: FlyingThings3D provides 21,818 training samples and 4,248 testing samples at 960×540 resolution.The dataset consists of rendered stereo image pairs and ground-truth disparity maps.
  • Sintel: Sintel is used as an additional testing dataset with ground-truth disparity maps.It is derived from an animated movie and provides a different synthetic evaluation setting.

E.2 TRAINING

The AutoDispNet-C experiments use a reduced search space and examine data augmentation and L2 regularization of the inner objective, while early stopping cannot be tested because required second-order derivatives are unavailable.

  • Training: AutoDispNet-C uses a smaller search space containing MaxPool3 × 3, SepConv3 × 3, and SkipConnect operations.Search images are downsampled by a factor of two and training runs for 300k mini-batch iterations.
  • Training: The experiments test data augmentation and L2 regularization applied to the network weights in the inner objective.
  • Training: Early stopping cannot be tested on AutoDispNet because custom feature-map correlation and resampling operations lack the required second-order derivatives for Hessian computation.
  • Training: Disparity-estimation augmentation includes spatial and appearance transformations, with strength varied by scaling transformation-distribution variances.Extracted networks are evaluated with the same augmentation parameters.

F RESULTS ON PENN TREEBANK

The Penn Treebank experiments assess stronger inner-objective L2 regularization for recurrent-cell search and report improved robustness as regularization increases.

  • Search setup: The PTB search uses a reduced recurrent-cell space with only ReLU and identity mapping as candidate operations.The remaining settings follow Liu et al. (2019).
  • Regularization: DARTS is run four times with different random seeds using L2 factors 5 × 10−7, 15 × 10−7, 45 × 10−7, and 135 × 10−7.The smallest factor is the DARTS default.
  • Results: Stronger inner-objective regularization makes the search procedure more robust, with median perplexity improving as the L2 factor increases.
  • Results: Figure 19 reports median test perplexity after training each discovered architecture from scratch for 1600 epochs.The figure also shows the validation perplexity of the search model as a blue dashed line.

G DISCOVERED CELLS ON SEARCH SPACES S1-S4 FROM SECTION 3 ON OTHER DATASETS

The appendix visualizes cells discovered across search spaces S1-S4 and datasets, comparing standard DARTS with DARTS-ES and showing AutoDispNet cells under different augmentation and L2 settings.

  • Standard DARTS: Figures 20, 29, and 30 show CIFAR-10 reduction and normal cells found by standard DARTS on spaces S1-S4 under default hyperparameters and different random seeds.
  • Standard DARTS: Figures 21 and 22 show normal and reduction cells found by standard DARTS on CIFAR-100 and SVHN across spaces S1-S4.The normal-cell caption highlights dominance of parameter-less operations such as skip connections and pooling.
  • DARTS-ES: Figures 24 and 25 show normal and reduction cells found by DARTS-ES on spaces S1-S4 using DARTS default hyperparameters.
Loading 1909.09656v2…