PaperlayerView exact revision on arXiv

Source-linked AI summary

Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search

Xiangxiang Chu, Tianbao Zhou, Bo Zhang, Jixiang Li

arXiv:1911.12126v4cs.LGcs.AIcs.CVstat.ML

TL;DR

DARTS can collapse because unfair advantages emerge under exclusive competition, producing excessive skip connections. Fair DARTS replaces that competition with independent operation weights and adds a zero-one loss to reduce discretization discrepancy. Experiments across two search spaces achieve state-of-the-art performance on CIFAR-10 and ImageNet.

  • Problem

    DARTS suffers performance collapse from excessive skip connections, while continuous architecture encodings can differ substantially from the discrete architectures derived from them.

  • Method

    Fair DARTS assigns each operation an independent architectural weight and uses a zero-one loss to push sigmoid weights toward 0 or 1.

  • Results

    Fair DARTS achieves state-of-the-art performance on CIFAR-10 and ImageNet, while Gaussian noise also produces competitive results by removing unfair advantage.

  • Takeaways & Limitations

    Breaking either unfair advantage or exclusive competition improves robustness, and collaborative competition reduces discretization discrepancy through polarized architectural weights.

  • Takeaways & Limitations

    Fair DARTS remains to be made more memory-friendly, and further methods for its approach remain to be explored.

Abstract

from arXiv · show

Differentiable Architecture Search (DARTS) is now a widely disseminated weight-sharing neural architecture search method. However, it suffers from well-known performance collapse due to an inevitable aggregation of skip connections. In this paper, we first disclose that its root cause lies in an unfair advantage in exclusive competition. Through experiments, we show that if either of two conditions is broken, the collapse disappears. Thereby, we present a novel approach called Fair DARTS where the exclusive competition is relaxed to be collaborative. Specifically, we let each operation's architectural weight be independent of others. Yet there is still an important issue of discretization discrepancy. We then propose a zero-one loss to push architectural weights towards zero or one, which approximates an expected multi-hot solution. Our experiments are performed on two mainstream search spaces, and we derive new state-of-the-art results on CIFAR-10 and ImageNet. Our code is available on https://github.com/xiaomi-automl/fairdarts .

1 Introduction

DARTS suffers performance collapse because skip connections gain an unfair advantage under exclusive competition. Fair DARTS makes operation weights independent, adds zero-one loss to reduce discretization discrepancy, and achieves state-of-the-art results on CIFAR-10 and ImageNet.

  • 1 Introduction: DARTS variants address excessive skip connections through dropout, hard limits, or early stopping, treating poor-performing models as impurities to filter out.These approaches intervene in training to prevent skip-connection aggregation or its associated high-curvature regime.
  • 1 Introduction: DARTS can select a shallow ImageNet model with 11 of 19 layers removed, achieving 66.4% top-1 accuracy.Fair DARTS instead infers a deeper model with only one layer removed and 75.6% top-1 accuracy.
  • 1 Introduction: The zero-one loss pushes sigmoid architectural weights toward 0 or 1, reducing the discrepancy between continuous encodings and discrete architectures.The method targets enabled-or-disabled operation choices instead of ambiguous intermediate weights.
  • 1 Introduction: The collapse results from two indispensable factors: unfair advantages and exclusive competition; breaking either factor makes the collapse disappear.Skip connections gain an advantage because they facilitate supernet training through a residual module that is broken in the outcome network.
  • 1 Introduction: Fair DARTS relaxes exclusive competition by assigning each operation an independent architectural weight, enabling collaborative competition.This allows skip connections and other operations to be selected together rather than forcing one-hot competition.
  • 1 Introduction: Experiments in two widely used search spaces show that Fair DARTS escapes performance collapse and achieves state-of-the-art networks on CIFAR-10 and ImageNet.The evaluation covers both proxy and proxyless settings.

2 Related Work

Related work extends DARTS to larger search spaces, stochastic or Gumbel-Softmax-based optimization, and robustness remedies. These approaches can still introduce bias or retain exclusive competition, whereas Fair DARTS avoids early stopping.

  • 2 Related Work: ProxylessNAS and FBNet apply DARTS in larger MobileNetV2-based search spaces.ProxylessNAS activates only two paths during supernet training, while FBNet uses differentiable Gumbel Softmax.
  • 2 Related Work: Activating only two ProxylessNAS paths updates some operations while leaving others unchanged, implicitly creating a bias.The approach assumes one path is best among all paths.
  • 2 Related Work: FBNet, SNAS, and GDAS use Gumbel Softmax to mimic one-hot encoding, whose exclusive competition risks exploitation by unfair advantages.The one-hot nature preserves the competitive structure implicated in DARTS collapse.
  • 2 Related Work: RobustDARTS uses early-stop regularization based on the largest eigenvalue, whereas Fair DARTS does not require early stopping.RobustDARTS links poor generalization to solutions with high validation-loss curvature and many skip connections.

3 The Downside of DARTS

DARTS relaxes architectural choices into softmax-weighted operations optimized through bi-level learning, but its search can favor skip connections and produce ambiguous continuous encodings. Experiments attribute collapse to unfair exclusive competition and show substantial discrepancy between continuous weights and the intended one-hot architecture.

  • DARTS background: DARTS represents cells as DAGs whose edges combine candidate operations through softmax-weighted outputs, then learns architecture coefficients and network weights with bi-level optimization.The search uses normal and reduction cells in S1 and a softmax-weighted summation of choices in S2.
  • Performance collapse: 5 out of 8 selected operations can be skip connections after 20 epochs in CIFAR-10 searches, indicating aggregation during DARTS optimization.This observation was obtained across four runs with different random seeds, selecting the top two operations for each of four intermediate nodes.
  • Performance collapse: 11 out of 19 layers become dominated by skip connections when DARTS is applied directly to ImageNet in S2.The dominant operation is defined as the one with the highest softmax(α) among all operations in a layer.
  • Cause of collapse: Skip connections progressively gain architectural weight after early boundary epochs while other operations decline, reflecting their unfair advantage in exclusive competition.The paper defines unfair advantage as an advantage contributing more to competition than to the resulting network’s performance.
  • Discretization discrepancy: The S1 softmax values range from above 0.17 to about 0.3, while an S2 layer approximates a one-hot choice with values such as [0.235, 0.057, 0.17, 0.016, 0.187, 0.269, 0.066].These narrow or non-dominant distributions make it difficult to distinguish selected operations confidently during discretization.
  • Discretization discrepancy: DARTS is often far from the one-hot representation assumed by its continuous relaxation, motivating a closer approximation between continuous and discrete encodings.The paper identifies this mismatch as a second insight alongside the unfair competition causing skip-connection aggregation.

4 Fair DARTS

Fair DARTS replaces softmax-based exclusive competition with independent sigmoid weights, then uses a differentiable zero-one loss to reduce the gap between continuous and discrete architectures.

  • 4.1 Stepping out the Pitfalls of Skip Connections: Fair DARTS applies sigmoid activation independently to each operation’s architectural weight, allowing operations to be switched on or off without suppression.This collaborative mechanism permits a multi-hot approximation rather than DARTS’s one-hot estimation.
  • 4.2 Resolve Discrepancy from Continuous Representation to Discrete Encoding: The zero-one loss pushes each sigmoid architectural weight toward 0 or 1 to minimize discretization discrepancy.Its design requires a maximum at 0.5, minima at 0 and 1, controlled gradients, and differentiability.
  • 4.2 Resolve Discrepancy from Continuous Representation to Discrete Encoding: The auxiliary loss is weighted by w_0−1 and combined with the architecture search objective during backpropagation.The objective seeks high accuracy while keeping continuous encodings close to their discrete counterparts.
  • 4.2 Resolve Discrepancy from Continuous Representation to Discrete Encoding: A negative-example loss illustrates why gradients must be controlled: constant gradients can rapidly push weights toward both ends after leaving 0.5.The proposed design instead uses smaller gradients near the midpoint and stronger attraction near the endpoints.
  • 4.2 Resolve Discrepancy from Continuous Representation to Discrete Encoding: Because independent edge weights converge toward extremities, Fair DARTS can infer submodels with a threshold instead of argmax.The zero-one loss is specific to the collaborative setting and cannot be directly applied under DARTS’s softmax competition.

5 Experiments and Results

Experiments evaluate Fair DARTS across CIFAR-10 and ImageNet search settings, including proxyless search, and report competitive architectures with reduced-resource configurations.

  • 5 Experiments and Results: Fair DARTS experiments use first-order optimization, with the search stage taking 10 GPU hours on a Tesla V100 in the reported proxy setting.The authors state that the same data processing and training tricks as DARTS and P-DARTS are used.
  • 5.1 Searching Architectures for CIFAR-10: 97.46% average top-1 accuracy is obtained on CIFAR-10 across seven FairDARTS searches with different random seeds.The smallest FairDARTS-a also reaches 97.46% accuracy with reduced parameters and multiply-adds.
  • 5.2 Transferring to ImageNet: 73.7% top-1 accuracy on ImageNet validation is achieved by FairDARTS-A using 3.6M parameters and 417M multiply-adds.FairDARTS-B reaches 75.1% in S1 with fewer parameters than comparable counterparts.
  • 5 Experiments and Results: Relaxing exclusive competition expands the search space from 7^19 models in ProxylessNAS to (2^7)^19 = 128^19 possible models.The paper characterizes this as about 18^19 times the ProxylessNAS search space.
  • 5.3 Searching Proxylessly on ImageNet: In proxyless ImageNet search, FairDARTS-D and C are illustrated using inverted bottleneck variants with and without inset skip connections and expansion.FairDARTS-D places two inverted bottleneck blocks near down-sampling and uses large kernels and large expansion blocks at the tail.

6 Ablation Study and Analysis

The ablations examine how removing unfair advantages, relaxing exclusive competition, and adding zero-one regularization affect DARTS collapse and discretization. They also compare these choices with dropout, early stopping, random noise, and constrained search spaces.

  • Unfair advantages: Removing skip connections from S1 prevents any operation, including parameter-free pooling operations, from suppressing the others.Across seven random seeds, the reduced space shows fair treatment under exclusive competition.
  • Zero-one loss: Without zero-one loss, sigmoid architectural weights span 0 to 0.6, forcing ambiguous discretization choices.The corresponding histogram covers a wide intermediate range rather than concentrating near binary endpoints.
  • Zero-one loss: Zero-one loss narrows sigmoid architectural weights toward 0 and 1, reducing the discrepancy between continuous encodings and discrete architectures.The loss is designed to make operations nearly disabled or completely enabled.
  • Zero-one loss: The selected auxiliary-loss weight is w_0−1 = 10 because it provides a trade-off between polarizing operations and avoiding domination of the entropy loss.Experiments vary w_0−1 from 0 to 16 while counting dominant operations.
  • Alternative remedies: Dropout reduces unfair advantage but requires tuning and, in P-DARTS, an additional prior limiting final-cell skip connections to M = 2.Early stopping avoids the stage where unfairness prevails, while Gaussian noise also disrupts unfairness and yields competitive results.
  • Alternative remedies: Randomly sampled models from the M = 2 regularized search space outperform DARTS, even without architecture search.The experiment averages seven models per group; Gaussian-noise experiments average four runs.
  • Discussion: Fair DARTS breaks exclusive competition and uses zero-one loss, while the authors leave broader applicability of the loss to existing approaches as future work.The discussion identifies this unresolved extension as a remaining issue.

7 Conclusion

The conclusion attributes DARTS skip-connection aggregation to unfair advantages combined with exclusive competition. Fair DARTS relaxes competition, polarizes architectural weights, and achieves state-of-the-art performance on CIFAR-10 and ImageNet.

  • Conclusion: Unfair advantages and exclusive competition are identified as the two indispensable factors behind excessive skip-connection aggregation.Breaking either factor improves robustness.
  • Conclusion: Fair DARTS lets each operation develop its architectural weight independently through collaborative competition.A zero-one auxiliary loss polarizes the weights to reduce discretization discrepancy.
  • Conclusion: Fair DARTS achieves state-of-the-art performance on both CIFAR-10 and ImageNet.The conclusion also reports competitive results when Gaussian noise removes unfair advantage.
  • Future work: Making the approach more memory-friendly is identified as future work.The authors suggest exploring alternatives related to Gumbel softmax.

1 Weight-sharing Neural Architecture Search

Weight-sharing neural architecture search reduces search overhead by training architectures through shared parameters. The paper distinguishes one-stage differentiable search from two-stage methods that use a trained supernet as an evaluator.

  • Overview: Weight-sharing methods are prominent because they reduce the enormous overhead of neural architecture search.They are broadly divided into one-stage and two-stage approaches.
  • One-stage approaches: One-stage approaches train a supernet and architecture-selection mechanism together, using reinforcement learning in ENAS or nested differentiable optimization in DARTS.DARTS creates exclusive competition between candidate operations.
  • Two-stage approaches: Two-stage approaches train a supernet first and then use it as an evaluator for sampled subnetworks.Unlike DARTS, some two-stage methods uniformly sample paths to give selected paths equal importance.

2 Search Spaces

The experiments use two mainstream search spaces: cell-based S1 for CIFAR-10 and block-level S2 for ImageNet. S1 uses a seven-node DAG, while S2 contains 19 layers with seven choices per layer.

  • Overview: The paper distinguishes cell-based search space S1 from block-level search space S2, using CIFAR-10 and ImageNet respectively.Proxy and proxyless indicate whether the search directly represents the backbone architecture.
  • Search Space S1: S1 follows DARTS with the zero operation excluded.Its cells are represented as directed acyclic graphs with flexible inner connections.
  • Search Space S1: S1 is a DAG of N = 7 nodes with two preceding-cell inputs, four intermediate nodes, and concatenated intermediate outputs.Each edge contains seven candidate operations.
  • Search Space S1: S1 candidate operations include pooling, skip connection, separable convolution, and dilated convolution choices.The listed convolution kernels are 3x3 and 5x5.
  • Search Space S2: S2 resembles ProxylessNAS, uses MobileNetV2 as its backbone, and contains L = 19 layers with N = 7 choices each.Its choices include inverted bottlenecks and skip connections.

3 Experiment Details

The experiments compare DARTS and Fair DARTS across search spaces, optimization settings, loss designs, and downstream detection, with Fair DARTS producing selectable, near-binary architectures.

  • Experimental scope: The paper evaluates DARTS and Fair DARTS in search spaces S1 and S2, with S1 corresponding to CIFAR-10 and S2 to ImageNet.The experiment inventory also includes architecture, dropout, optimization, loss, and object-detection studies.
  • Fair DARTS behavior: Fair DARTS permits substantial skip connections while architectural weights coexist near zero or one, reducing the discretization gap.This combines collaborative competition with near-binary architectural encodings.
  • Searched architectures: FairDARTS-a activates only three nodes in its normal cell and uses more reduction-cell edges to compensate for down-sampling information loss.Because reduction cells are a small proportion of the architecture, the resulting model remains lightweight.
  • Loss comparison: The zero-one loss makes Fair DARTS less sensitive to initialization, while sigmoid values approach their optima more slowly than under L′0−1.The loss also gives operations a fair opportunity near σ(α)=0.5.
  • Optimization comparison: Under bi-level optimization, a 5 × 5 separable convolution on edge (2, 2) decreases substantially after weighing higher early, indicating robustness to a local optimum.The comparison is illustrated in Fig. 3.
  • Object detection: On MS COCO, Fair DARTS backbones used as RetinaNet replacements achieve 31.9% average precision.The models are trained and evaluated for 12 epochs with comparable mobile backbones.

4 Figures

The figures track architectural-weight evolution and searched-cell structure, showing how Fair DARTS avoids skip-connection aggregation and approaches binary operation selection.

  • DARTS with zero-one loss: Adding the zero-one loss to DARTS with softmax worsens skip-connection aggregation because weak operations are driven toward zero in a domino effect.The inferred models reach 96.77 ± 0.29% on CIFAR-10, not far from original DARTS.
  • Final architectures: Fair DARTS heatmaps show architectural weights mainly near 0 and 1, while searched FairDARTS-a and FairDARTS-b cells illustrate the resulting architectures.The figures attribute the near-binary values to sigmoid features and the auxiliary zero-one loss.
  • Trajectory comparisons: The figures separately compare sigmoid evolution under L′0−1 and under single-level versus bi-level optimization.These visualizations provide the trajectory comparisons summarized in the experiment discussion.
  • DARTS evolution: DARTS softmax evolution shows selected skip connections gradually suppressing other operations through unfair advantage.The suppression occurs on specific normal- and reduction-cell edges.
  • Fair DARTS evolution: Fair DARTS sigmoid evolution gives skip connections equal opportunity under collaborative competition, allowing them to coexist with other operations.This is visible on selected normal-cell edges and nearly all reduction-cell edges.
  • Ablation: Without skip connections, DARTS operations are encouraged to demonstrate their individual strengths after the unfair advantage is removed.The corresponding evolution is shown for CIFAR-10 in S1.
Loading 1911.12126v4…