PaperlayerView exact revision on arXiv

Source-linked AI summary

OStr-DARTS: Differentiable Neural Architecture Search based on Operation Strength

Le Yang, Ziwei Zheng, Yizeng Han, Shiji Song, Gao Huang, Fan Li

arXiv:2409.14433v1cs.AI

TL;DR

DARTS can produce deteriorating architectures when magnitude-based selection does not reflect operation contributions, while prior work has focused mainly on supernet optimization. OStr-DARTS selects operations by their effect on the loss, and the paper reports that this addresses degeneration without modifying supernet optimization.

  • Problem

    DARTS can yield deteriorating architectures when architecture parameters fail to reflect operation contributions, while most prior work focuses on supernet optimization rather than selection.

    Source: §I · p. 1

  • Method

    OStr-DARTS selects operations using operation strength, which estimates their contribution to the supernet through their effect on the final loss.

    Source: §I · p. 2

  • Results

    The authors report that operation-strength selection effectively addresses DARTS degeneration without modifying supernet optimization.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors conclude that magnitude-based architecture selection can be a main reason for DARTS instability.

    Source: §I · p. 2, §VI · p. 12

    Operation strength uses a first-order Taylor expansion because calculating the Hessian matrix for a second-order estimate is costly.

    Source: §VI · p. 12

Abstract

from arXiv · show

Differentiable architecture search (DARTS) has emerged as a promising technique for effective neural architecture search, and it mainly contains two steps to find the high-performance architecture: First, the DARTS supernet that consists of mixed operations will be optimized via gradient descent. Second, the final architecture will be built by the selected operations that contribute the most to the supernet. Although DARTS improves the efficiency of NAS, it suffers from the well-known degeneration issue which can lead to deteriorating architectures. Existing works mainly attribute the degeneration issue to the failure of its supernet optimization, while little attention has been paid to the selection method. In this paper, we cease to apply the widely-used magnitude-based selection method and propose a novel criterion based on operation strength that estimates the importance of an operation by its effect on the final loss. We show that the degeneration issue can be effectively addressed by using the proposed criterion without any modification of supernet optimization, indicating that the magnitude-based selection method can be a critical reason for the instability of DARTS. The experiments on NAS-Bench-201 and DARTS search spaces show the effectiveness of our method.

I. INTRODUCTION

DARTS makes neural architecture search more efficient through gradient-based supernet optimization, but its magnitude-based operation selection can produce unstable, deteriorating architectures. OStr-DARTS instead ranks operations by their effect on the loss, addressing degeneration without changing supernet optimization.

A. NAS and DARTS

  • Earlier NAS approaches based on reinforcement learning and evolutionary algorithms commonly required hundreds of GPU days, motivating more efficient search methods.One-shot methods use weight sharing, and DARTS enables gradient-based architecture search.Source: §I · p. 1

I. INTRODUCTION

  • DARTS jointly optimizes model weights and architecture parameters in a mixed-operation supernet, then selects operations with the largest architecture parameters.This second step is the magnitude-based selection method.Source: §I · p. 1, Figure 1 · p. 2
  • Magnitude-based selection can fail when architecture parameters misrepresent operation contributions, including because parameter coupling is ignored and skip connections can gain large weights.These problems can cause the final architecture’s performance to collapse.Source: §I · p. 1

A. NAS and DARTS

  • Prior DARTS work mainly changes supernet optimization, while leaving the final architecture selection process largely unaddressed.The paper frames operation selection as a discrete problem that most optimization methods cannot effectively solve.Source: §I · pp. 1–2

I. INTRODUCTION

  • Operation strength estimates an operation’s contribution by its effect on the final loss and selects the highest-strength operations to build the final architecture.The resulting method, OStr-DARTS, addresses degeneration without modifying supernet optimization.Source: §I · p. 2

B. Architecture Selection for DARTS

DARTS architecture selection should identify the operation that contributes most to the supernet, rather than simply choosing the largest architecture parameter. The section frames selection as a discrete problem and proposes estimating each candidate’s effect on loss when selected.

B. Architecture Selection for DARTS

  • Perturbation-based selection can require retraining to convergence, while performance predictors require specialized models and encodings; pruning-based ranking is limited to smaller search spaces.The cited pruning-based NAS methods achieve only fair performance, and the perturbation approach evaluates edges one by one.Source: B. Architecture Selection for DARTS · pp. 2–3

III. METHOD

  • OStr-DARTS estimates each operation’s importance by the loss change induced by selecting it, then chooses the highest-strength operation on each edge.The optimized supernet can use classical DARTS optimization or an improved method such as PC-DARTS.Source: §III · p. 3

A. Formulation of DARTS

  • DARTS represents a cell as a directed acyclic graph with candidate operations on each edge, but the largest architecture parameter need not identify the operation contributing most.At selection, each mixed operation is replaced by the operation with the largest α value in the standard magnitude-based approach.Source: A. Formulation of DARTS · p. 3

B. Rethinking selection problem of DARTS

  • The discrete selection problem requires one operation per edge, and its complexity motivates using operation effects on loss as a proxy for finding a good solution.The pruning analogy motivates scoring operations by their impact on the objective, but the selection criterion must account for selecting rather than removing an operation.Source: B. Rethinking selection problem of DARTS · pp. 3–4
  • Architecture selection chooses the most important operation per edge, whereas pruning removes unimportant weights; applying pruning-style removal to selection can yield a suboptimal architecture.The proposed estimate measures the loss change from substituting a candidate operation’s features for the mixed-operation features.Source: B. Rethinking selection problem of DARTS · p. 4

C. Operation Strength

Operation strength scores an operation by the absolute change in final loss when it replaces the mixed operation. The paper relates this score to architecture-parameter gradients and selects the highest-scoring operation on each edge.

C. Operation Strength

  • Operation strength is the absolute final-loss change when an optimized supernet selects a candidate operation in place of the mixed operation.The architecture selects, on each edge, the operation with the largest strength.Source: C. Operation Strength · p. 5
  • Operation strength can be obtained directly from architecture-parameter gradients already computed during optimization, without extra computational cost.The formulation extends to multiple cells using the corresponding mixed-operation and candidate-operation features.Source: C. Operation Strength · p. 5
  • At an ideal supernet optimum, architecture-parameter gradients tend toward zero, so the expected operation strength also tends toward zero.The paper describes the practical estimate as remaining nonzero because it is based on stochastic gradient evaluations.Source: C. Operation Strength · p. 5
  • Because stochastic gradient evaluations make practical strength nonzero, the method averages absolute gradient magnitudes over B minibatches.For batch size 1, the strength tends toward the expected absolute gradient magnitude and is proportional to gradient standard deviation under the stated i.i.d. assumption [57].Source: C. Operation Strength · p. 5
  • Algorithm 1 updates network weights and architecture parameters, estimates operation strengths, and selects an architecture from those scores.When enabled, early stopping halts the search after the selected architecture remains unchanged for C epochs, subject to the search-epoch limit T.Source: C. Operation Strength · p. 5

D. OStr-DARTS

OStr-DARTS uses operation-strength gradients during supernet search to select operations without extra computation. It can use classical or improved DARTS optimization and optionally stop early when the architecture stabilizes.

D. OStr-DARTS

  • OStr-DARTS can use classical bi-level optimization or an improved procedure such as PC-DARTS.Operation-strength gradients are available during optimization, so the method estimates candidate strengths without extra costs.Source: D. OStr-DARTS · p. 5
  • The search can stop early when the selected architecture remains unchanged for a specified number of epochs.The stopping criterion is optional; the algorithm also has a maximum search-epoch limit.Source: D. OStr-DARTS · p. 5
  • The paper reports that OStr-DARTS architectures converge quickly in some search spaces, which it presents as further evidence of stability.Early stopping is tied to architecture stability during search.Source: D. OStr-DARTS · p. 5

A. Avoid the domination of skip connections

Operation strength counterbalances skip connections’ larger architecture parameters by also measuring how distinctive their features are and how much they affect the loss. This can retain useful skip connections while excluding redundant ones.

A. Avoid the domination of skip connections

  • Skip connections can receive larger architecture parameters because the mixed input is closer in variance to the optimal feature map than a convolution output.The resulting parameter advantage can cause skip-connection domination in a well-optimized supernet.Source: A. Avoid the domination of skip… · p. 6
  • A skip connection’s smaller difference between its output and the mixed output lowers its operation strength, offsetting its larger architecture parameter.These two terms balance and avoid skip-connection domination in OStr-DARTS.Source: A. Avoid the domination of skip… · p. 6
  • Operation strength balances architecture-parameter magnitude with residual-feature distinctiveness, so a large parameter alone does not determine operation importance.The residual-feature norm reflects how distinctive an operation’s generated features are relative to those from other operations.Source: A. Avoid the domination of skip… · p. 6
  • The analysis preserves important skip connections while eliminating redundant ones, and Fig. 5(b) shows the resulting optimal cell architecture.On Edge-3 and Edge-5, skip connections have larger β values, but operation strength can ignore those with little effect on the final loss.Source: Figure 5 · p. 5, A. Avoid the domination of skip… · p. 6, Figure 6 · p. 6

B. Operation strength: architecture parameter perspective

Operation strength estimates an operation’s importance by the change in loss when that operation is selected. The paper approximates this loss change using a directional derivative.

B. Operation strength: architecture parameter perspective

  • The loss difference is evaluated by setting the target operation’s architecture parameter to positive infinity, which makes its softmax weight approach one.The resulting loss-change estimate is approximated using the direction associated with selecting that operation.Source: B. Operation strength: architecture… · pp. 6–7
  • Operation strength estimates an operation’s importance for the discrete architecture choice by measuring its effect on the loss when selected.The proposed approximation uses a directional derivative of the loss to estimate the change associated with selecting the operation.Source: B. Operation strength: architecture… · pp. 6–7

C. The novelty and superiority of operation strength

The paper argues that operation strength better targets the discrete architecture-selection problem than magnitude-based selection or pruning indicators. It also introduces OStr-DARTS*, a Taylor-expansion variant that performs promisingly across the evaluated search spaces.

C. The novelty and superiority of operation strength

  • Operation strength is presented as a closer approximation to the discrete selection problem than magnitude-based conversion from a continuous solution.The authors argue this avoids an instability-prone discrete step and can select important skip connections while eliminating redundant ones.Source: C. The novelty and superiority of… · p. 7
  • Pruning indicators estimate importance by removing weights or operations, whereas architecture selection must choose an operation and set its mixture weight to one.The authors argue that directly applying pruning indicators to architecture selection can yield suboptimal results.Source: C. The novelty and superiority of… · p. 7

D. OStr-DARTS*

  • OStr-DARTS* uses a new operation-strength criterion based on a Taylor expansion, and experiments report better performance than the Naive variant.The new criterion uses a different expansion formulation for estimating loss change.Source: D. OStr-DARTS* · p. 7
  • Both OStr-DARTS variants achieve promising performance on NAS-Bench-201 and DARTS, while a near-flat gradient can make the OStr-DARTS loss-change estimate less accurate.The variants differ in their Taylor-expansion points.Source: D. OStr-DARTS* · p. 7

A. Experiments on NAS-Bench-201 search space

On NAS-Bench-201, OStr-DARTS methods outperform related NAS methods and find architectures close to the search-space optimum. The reported search stabilizes quickly, and the paper also evaluates the methods in the DARTS search space.

2) Comparison results.:

  • OStr-DARTS methods outperform related NAS methods and find almost the best architecture across three NAS-Bench-201 datasets.Across five random-seed runs, OStr-DARTS finds the same optimal solution each time, with performance close to the search-space optimum.Source: 2) Comparison results · p. 7, Table I · p. 8
  • After around 25 CIFAR-10 search epochs, the resulting architecture no longer changes.The paper reports this as an example of the search process reaching its optimal point quickly.Source: 2) Comparison results · p. 8
  • OStr-DARTS achieves higher accuracy and lower standard deviation than OStr-DARTS* on three datasets.The authors report that including architecture-parameter magnitude in the selection criterion benefits selection.Source: 2) Comparison results · p. 8

B. Performance on DARTS search space

  • The DARTS-space evaluation uses CIFAR-10, CIFAR-100, and ImageNet, with ImageNet inference limited to 600M FLOPs.The evaluated architectures are searched on CIFAR-10 and trained from scratch for evaluation; the paper also includes an augmented training setting using RandAugmentation and double epochs.Source: B. Performance on DARTS search space · p. 8, 1) Implementation Details · p. 8

2) Results on CIFAR-10 and CIFAR-100:

OStr-DARTS improves DARTS-based architecture selection results across CIFAR and ImageNet evaluations, and its selection criteria are also examined for stability and degeneration.

2) Results on CIFAR-10 and CIFAR-100:

  • DARTS outperforms all evaluated methods in Table II on CIFAR-10 and CIFAR-100; with RandAugmentation, OStr-DARTS reaches 98.02% and 85.30% accuracy on the respective datasets.Source: 2) Results on CIFAR-10 and CIFAR-100 · p. 9

3) Results on ImageNet:

  • 76.2% and 75.7% test accuracy: architectures searched on CIFAR-10 with OStr-DARTS and OStr-DARTS*, respectively, rank among popular NAS methods on ImageNet.The passage also reports that OStr-PC-DARTS is significantly superior to PC-DARTS on ImageNet, without specifying the values here.Source: Table III · p. 9, 3) Results on ImageNet · p. 9

4) Direct search on ImageNet:

  • 76.3% top-1 accuracy: direct ImageNet search with OStr-DARTS outperforms the tested baselines; OStr-PC-DARTS reaches 76.2% versus PC-DARTS at 75.8% with the same search costs.With augmented training, OStr-DARTS and OStr-PC-DARTS reach 77.1% and 76.7%, respectively.Source: Table IV · p. 10, 4) Direct search on ImageNet · p. 10

C. Ablation Study

  • The study evaluates selected-architecture accuracy across epochs, method visualizations, and the relationship between selection indicators and standalone performance.The listed figures cover DARTS and OStr-DARTS accuracy on CIFAR-10/100, three selection criteria in NAS-Bench-201, and visualizations of DARTS, OStr-DARTS, Naive, and OStr-DARTS*.Source: C. Ablation Study · p. 10, Figure 10 · p. 10, Figure 9 · p. 10

1) The importance of architecture selection:

The ablations find that operation-strength selection addresses degeneration without changing supernet optimization, while showing more stable search and stronger indicator–accuracy correlations.

1) The importance of architecture selection:

  • Selected architectures do not lose performance as search epochs increase when operation strength or the tested alternatives guide selection, supporting selection criteria as a source of classical DARTS instability.The compared criteria use the same optimized supernet; the authors identify magnitude-based selection as a possible critical reason for instability.Source: 1) The importance of architecture… · p. 10

2) Optimization stability:

  • 0.06 and 0.08: the proposed methods’ reported standard deviations are lower than DARTS, whose accuracy standard deviation across search epochs can reach 0.37.DARTS accuracy begins to drop after 25 epochs, while OStr-DARTS is reported to address degeneration and stabilize after around 25 epochs.Source: Table VI · p. 11, 2) Optimization stability · p. 11

3) Stand-Alone Model Performance:

  • 0.07 versus 0.93: in the DARTS search space, architecture-parameter rank correlation with standalone accuracy is weak, while operation-strength correlation is much higher.NAS-Bench-201 shows the same qualitative pattern: operation strength correlates more strongly with standalone accuracy, and the highest-strength operations correspond to the most accurate standalone models.Source: Figure 13 · p. 11, 3) Stand-Alone Model Performance · p. 11, Figure 12 · p. 11

D. Transferring to Object Detection

OStr-DARTS performs strongly when transferred to object detection, while the paper qualifies its method with assumptions about estimating discrete-selection loss changes and computational limits on the Taylor approximation.

D. Transferring to Object Detection

  • 34.4% AP: OStr-DARTS achieves the highest average precision among the evaluated methods on RetinaNet object detection using MS-COCO.The authors suggest the results indicate that OStr-DARTS can maintain generalization when transferred to tasks such as object detection.Source: Table VII · p. 12, D. Transferring to Object Detection · p. 12

VI. LIMITATIONS AND FUTURE WORKS

  • Optimality depends on operation strength accurately estimating loss changes for the discrete NAS selection problem, an assumption the authors say needs further study.The method uses a first-order Taylor expansion because calculating the Hessian for a second-order term is computationally costly.Source: §VI · p. 12

VII. CONCLUSION

  • The conclusion attributes poor DARTS generalization to magnitude-based selection rather than entirely to supernet optimization, while leaving optimization improvements available for combination with OStr-DARTS.The authors report consistent exploration of better-performing architectures across several search spaces and datasets.Source: §VI · p. 12, §VII · p. 12

Related papers

Loading 2409.14433v1…