PaperlayerView exact revision on arXiv

Source-linked AI summary

PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search

Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, Hongkai Xiong

arXiv:1907.05737v4cs.CVcs.LG

TL;DR

DARTS makes architecture search fast but incurs substantial memory and computation overhead when training a super-network. PC-DARTS samples channels for operation search, bypasses the rest, and uses edge normalization to stabilize connectivity selection. It achieves 2.57% CIFAR10 error in 0.1 GPU-days and 24.2% ImageNet top-1 error in 3.8 GPU-days.

  • Problem

    DARTS suffers from memory inefficiency because outputs for all candidate operations must be stored at each node, forcing smaller search batches that can slow and destabilize search.

  • Method

    PC-DARTS samples 1/K of channels for operation mixtures, bypasses the remaining channels, and adds edge normalization to reduce sampling-induced connectivity uncertainty.

  • Results

    PC-DARTS achieves 2.57% CIFAR10 error in 0.1 GPU-days and 24.2% ImageNet top-1 error in 3.8 GPU-days.

  • Takeaways & Limitations

    Partial channel connections reduce search memory consumption, enable larger batches, and combine with edge normalization to improve search efficiency and stability.

  • Takeaways & Limitations

    On CIFAR10, all evaluated algorithms fail to gain accuracy in enlarged search spaces because the dataset is relatively simple and searched-architecture performance appears to saturate.

Abstract

from arXiv · show

Differentiable architecture search (DARTS) provided a fast solution in finding effective network architectures, but suffered from large memory and computing overheads in jointly training a super-network and searching for an optimal architecture. In this paper, we present a novel approach, namely, Partially-Connected DARTS, by sampling a small part of super-network to reduce the redundancy in exploring the network space, thereby performing a more efficient search without comprising the performance. In particular, we perform operation search in a subset of channels while bypassing the held out part in a shortcut. This strategy may suffer from an undesired inconsistency on selecting the edges of super-net caused by sampling different channels. We alleviate it using edge normalization, which adds a new set of edge-level parameters to reduce uncertainty in search. Thanks to the reduced memory cost, PC-DARTS can be trained with a larger batch size and, consequently, enjoys both faster speed and higher training stability. Experimental results demonstrate the effectiveness of the proposed method. Specifically, we achieve an error rate of 2.57% on CIFAR10 with merely 0.1 GPU-days for architecture search, and a state-of-the-art top-1 error rate of 24.2% on ImageNet (under the mobile setting) using 3.8 GPU-days for search. Our code has been made available at: https://github.com/yuhuixu1993/PC-DARTS.

1 INTRODUCTION

PC-DARTS reduces the memory and computation burdens of differentiable architecture search by sampling channels during operation search and stabilizing connectivity selection with edge normalization. This enables larger batches, faster search, and strong CIFAR10 and ImageNet results.

  • 1 INTRODUCTION: PC-DARTS randomly samples a subset of channels for operation selection while bypassing the remaining channels through a shortcut.The sampled channels serve as a surrogate for computation over all channels.
  • 1 INTRODUCTION: Edge normalization shares learned edge-selection hyper-parameters across training to reduce connectivity instability caused by changing sampled channels.The architecture therefore becomes less sensitive to which channels are sampled across iterations.
  • 1 INTRODUCTION: 2.57% CIFAR10 error was achieved in less than 0.1 GPU-days, versus 2.76% for DARTS requiring 1.0 GPU-day.The search used a single Tesla V100 GPU and took around 1.5 hours.
  • 1 INTRODUCTION: 24.2% ImageNet top-1 error was achieved under the mobile setting using 3.8 GPU-days of search.PC-DARTS also enabled direct ImageNet search, whereas DARTS failed due to low stability.

2 RELATED WORK

NAS develops methods for exploring large architecture spaces, progressing from evolutionary and weight-sharing approaches toward one-shot and differentiable search. PC-DARTS is presented within this trajectory as a partially connected approach illustrated for information propagation to a network node.

  • 2 RELATED WORK: NAS seeks automatic network architectures by exploring a large architecture space under training-data and deployment constraints.The field aims to replace conventional handcrafted architecture design.
  • 2 RELATED WORK: Evolutionary NAS methods apply genetic operations to evolve individual architectures or architecture families toward better performance.Examples include hierarchical network representations and gene-based architecture descriptions.
  • 2 RELATED WORK: One-shot architecture search trains an over-parameterized super-network covering candidate operations once, reducing the cost of evaluating searched candidates.Earlier approaches shared weights between searched and newly generated networks.
  • 2 RELATED WORK: Figure 1 illustrates PC-DARTS information propagation to node #3 and identifies two sets of search hyper-parameters.The supplied caption introduces the example but does not specify the figure's axes, rows, columns, colors, or comparison outcome.
  • 2 RELATED WORK: DARTS combines architecture search and evaluation through differentiable optimization of a super-network, but prior work identified instability as a drawback.The strongest sub-network is preserved after search and then retrained.

3 THE PROPOSED APPROACH

PC-DARTS reduces DARTS’s memory and computation burden by applying operation search to sampled channels while bypassing the remainder. Edge normalization addresses connectivity instability caused by channel sampling, improving search efficiency and stability.

  • DARTS stores outputs from all candidate operations on each edge, creating memory inefficiency that forces smaller search batches and can harm speed, stability, and accuracy.
  • Partial channel connections: PC-DARTS sends a sampled subset of channels through mixed operations and directly bypasses the remaining channels via a shortcut.
  • Partial channel connections: 1/K of the channels are selected, reducing memory overhead by K times and enabling a K times larger search batch size.
  • Partial channel connections: Partial channel connections regularize operation selection by reducing bias toward weight-free operations such as skip-connect and pooling.
  • Edge normalization: Random channel sampling can destabilize edge connectivity because architecture parameters are optimized using different sampled channels across iterations.
  • Edge normalization: Edge normalization adds shared edge-level parameters β_i,j, making the selected architecture less sensitive to sampled channels and adding negligible computation overhead.
  • Discussion and relationship to prior work: PC-DARTS combines channel sampling and edge normalization, achieving the best reported ImageNet accuracy in the DARTS search space with 3.8 GPU-days of search cost.

4.1 DATASETS AND IMPLEMENTATION DETAILS

The experiments evaluate PC-DARTS on CIFAR10 and ImageNet using a separate search stage followed by training the selected architecture from scratch. The implementation retains the conventional eight-operation search space and uses channel shuffling for a faster partial-connection implementation.

  • Datasets: The evaluation uses CIFAR10 and ImageNet, two benchmark datasets for neural architecture search.
  • Implementation details: Architecture search is performed separately, after which the selected architecture is trained from scratch.
  • Implementation details: The search training set is partitioned so network parameters and architecture hyper-parameters are optimized using different subsets.
  • Results: 2.57% test error is reported on CIFAR10 after selecting the best of five PC-DARTS runs using standalone validation.
  • Implementation details: The conventional operation space contains eight choices, including separable and dilated convolutions, pooling, skip-connect, and zero.
  • Implementation details: The efficient implementation uses the first 1/K channels for operation mixing and shuffles output channels before subsequent computation.

4.2 RESULTS ON CIFAR10

On CIFAR10, PC-DARTS uses partial channel sampling to reduce search cost while retaining strong accuracy. With K = 4, the larger batch size accelerates search, yielding a 2.57% error rate in 0.1 GPU-days.

  • Search setup: 1/4 channel sampling increases the CIFAR10 search batch size from 64 to 256.The setting samples only 1/4 of the features on each edge.
  • Search efficiency: 3 hours on GTX 1080Ti or 1.5 hours on Tesla V100 makes PC-DARTS almost 4× faster than first-order DARTS.The increased batch size enables this shorter search time.
  • Searched cells: ImageNet search produces a deeper normal cell, while its reduction cell remains very similar to the CIFAR10 reduction cell.The searched cells are visualized in Figure 2 after CIFAR10 evaluation.
  • Accuracy and comparison: 2.57% error in 0.1 GPU-days establishes PC-DARTS as a fast CIFAR10 method below 3% error.The reported result is compared with recent architecture-search approaches, including DARTS, ProxylessNAS, and P-DARTS.

4.3 RESULTS ON IMAGENET

On ImageNet, PC-DARTS adapts partial channel sampling and evaluates architectures in the mobile setting. It reports 24.2% top-1 error for direct ImageNet search and compares sampling rates through search cost and accuracy.

  • Search setup: 1/2 channel subsampling is used on ImageNet because the dataset is more difficult than CIFAR10.The paper states that the higher sampling rate preserves more information.
  • ImageNet results: 24.2% top-1 error and 7.3% top-5 error are reported for the architecture searched directly on ImageNet.The result is described as the best known performance to date in the mobile setting.
  • ImageNet results: 25.1% top-1 error and 7.8% top-5 error are obtained when the architecture searched on CIFAR10 is evaluated on ImageNet.These values outperform DARTS at 26.7% top-1 and 8.7% top-5 error.
  • Sampling-rate analysis: Sampling rates 1/1, 1/2, 1/4, and 1/8 are compared by search cost and accuracy, with 1/4 identified as the efficiency–accuracy tradeoff.The Figure 3 comparison reports a dramatic accuracy drop at 1/8.

4.4 ABLATION STUDY

The ablations examine channel-sampling efficiency, component contributions, and search stability. PC-DARTS balances accuracy and efficiency while edge normalization substantially reduces instability across runs and evaluation settings.

  • Effectiveness of channel proportion 1/K: A 1/4 sampling rate outperforms 1/2 and 1/1 in both search time and CIFAR10 accuracy, while 1/8 further reduces computation with an accuracy tradeoff.The experiments compare sampling rates of 1/1, 1/2, 1/4, and 1/8.
  • Effectiveness of channel proportion 1/K: The sampling-rate results reveal redundancy in super-network optimization and a gap between better super-network optimization and better searched architectures.The authors interpret channel sampling as regularization against overfitting the super-network.
  • Contributions of different components of PC-DARTS: Edge normalization stabilizes architecture search: without it, CIFAR10 testing error ranges from 2.54% to 3.01%, whereas five runs differ by at most 0.15% with it.Edge normalization also cooperates with partial channel connections and adds very little computational cost.
  • Stability of our approach: Across five independent runs, PC-DARTS has lower variability than DARTS-v1 and DARTS-v2, with standard deviations of ±0.07%, ±0.15%, and ±0.21%, respectively.The runs use the same hyperparameters but different random seeds.
  • Stability of our approach: PC-DARTS is more robust to changes in search epochs and enlarged search spaces; unlike DARTS-v2, it mostly preserves performance when the cell grows from 6 to 7 nodes.All three algorithms improve from 5 to 6 nodes, but DARTS-v2 suffers a significant drop from 6 to 7 nodes.
  • Stability of our approach: On CIFAR10, enlarged search spaces do not improve accuracy because the dataset is relatively simple and searched-architecture performance appears to saturate.This scope boundary applies to the reported enlarged-search-space experiments.
  • Stability of our approach: The reported robustness is attributed to regularization that adapts search to dynamic architectures and reduces the pruning gap caused by the none operator.This interpretation is presented as the explanation for PC-DARTS robustness across evaluation scenarios.

4.5 TRANSFERRING TO OBJECT DETECTION

The ImageNet architecture found by PC-DARTS transfers to object detection through an SSD backbone. On MS-COCO, it achieves competitive average precision with substantially fewer FLOPs than the compared SSD configurations.

  • Transfer to object detection: Using the ImageNet-searched backbone, SSD achieves 28.9% AP with 1.2B FLOPs on MS-COCO test-dev 2015.The model is trained on the MS-COCO trainval dataset using the stated standard split pipeline.
  • Transfer to object detection: The result is 5.7% higher AP than SSD300 with 29× fewer FLOPs and 2.1% higher AP than SSD512 with 83× fewer FLOPs.These comparisons use the reported SSD baselines.
  • Transfer to object detection: PC-DARTS surpasses the strongest compared Lite detector, SSDLiteV3, by 6.9% AP.The paper presents this as evidence that classification gains transfer to the more challenging object-detection task.

5 CONCLUSIONS

PC-DARTS reduces architecture-search memory and computation by sampling channels, enabling larger batches and greater stability; edge normalization provides additional stabilization. The method searches quickly and reports strong classification accuracy, while the authors emphasize regularization and batch size as useful directions for stabilizing differentiable NAS.

  • Conclusions: PC-DARTS randomly samples channels for operation search, bypasses the remainder, and uses a lightweight edge-normalization module to improve search stability.The partial connection reduces memory use and enables a larger batch size.
  • Conclusions: PC-DARTS completes search in 0.1 GPU-days on CIFAR10 or 3.8 GPU-days on ImageNet and reports state-of-the-art classification accuracy, particularly on ImageNet.These are the paper's reported search budgets and classification outcome.
  • Conclusions: The authors identify regularization and larger batch sizes as factors that can benefit differentiable architecture search stability.They present PC-DARTS as one efficient way to incorporate both factors in a single pipeline.
  • Conclusions: The experiments reveal a gap between improving super-network optimization and finding a better architecture, with regularization described as helping shrink that gap.The authors frame this as an insight for future differentiable architecture-search research.
Loading 1907.05737v4…