Source-linked AI summary
Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
Xin Chen, Lingxi Xie, Jun Wu, Qi Tian
TL;DR
Differentiable NAS can search efficiently, but shallow search and deep evaluation create a depth gap that may favor unsuitable operations. P-DARTS progressively increases search depth, using search-space approximation and regularization to control overhead and instability. It reports strong results on CIFAR10, CIFAR100, and ImageNet, including 2.50% CIFAR10 test error and 24.4%/7.4% ImageNet top-1/5 errors.
Problem
DARTS searches in shallow networks but evaluates in deeper ones, so operations favored during search may differ from those preferred for deep evaluation.
Method
P-DARTS progressively increases candidate-architecture depth and uses search space approximation and regularization to reduce overhead and instability.
Results
P-DARTS achieves 2.50% CIFAR10 test error with 3.4M parameters and 24.4%/7.4% ImageNet top-1/5 errors, while reaching 3% CIFAR10 error in 0.3 GPU-days.
Takeaways & Limitations
The approach achieves superior performance on both proxy and target datasets while providing a fast NAS procedure.
Takeaways & Limitations
Reusing parameters from the previous search stage lowers accuracy because it biases search toward architectures suited to shallow networks rather than deep ones.
Abstract
from arXiv · showhide
Recently, differentiable search methods have made major progress in reducing the computational costs of neural architecture search. However, these approaches often report lower accuracy in evaluating the searched architecture or transferring it to another dataset. This is arguably due to the large gap between the architecture depths in search and evaluation scenarios. In this paper, we present an efficient algorithm which allows the depth of searched architectures to grow gradually during the training procedure. This brings two issues, namely, heavier computational overheads and weaker search stability, which we solve using search space approximation and regularization, respectively. With a significantly reduced search time (~7 hours on a single GPU), our approach achieves state-of-the-art performance on both the proxy dataset (CIFAR10 or CIFAR100) and the target dataset (ImageNet). Code is available at https://github.com/chenxin061/pdarts.
1. Introduction
Neural architecture search reduces manual design costs, but differentiable methods face a depth gap because shallow search can favor operations that do not suit deep evaluation. P-DARTS progressively increases search depth and uses approximation and regularization to address the resulting overhead and instability.
- Neural architecture search automates neural-network design for image recognition, replacing parts of handcrafted architecture engineering.
- Hundreds or even thousands of GPU-days made reinforcement-learning and evolutionary NAS pipelines barely affordable.
- DARTS reduces search cost to a few GPU-days but searches shallow networks and evaluates deeper ones, creating a depth gap.
- P-DARTS progressively increases network depth across search stages to bridge the gap between search and evaluation.
- Search space approximation reduces candidate operations as depth grows, while search space regularization addresses instability from skip-connect dominance.
- 2.50% test error on CIFAR10 with 3.4M parameters, and 24.4%/7.4% top-1/5 errors on ImageNet, demonstrate strong proxy and target-dataset performance.The reported CIFAR100 result is 15.92% test error with 3.6M parameters.
2. Related Work
NAS evolved from expensive methods that searched complete networks or cells toward more efficient differentiable approaches. This progression reduced computational cost while addressing consistency and proxy-task limitations.
- Early NAS searched complete network topologies, whereas later methods focused on reusable cells assembled into task-specific networks.
- Evolutionary and reinforcement-learning NAS achieved strong image-recognition performance but required 3,150 and 1,800 GPU-days, respectively.
- PNAS used a surrogate model for a 5× speedup over NASNet, while ENAS reduced search cost to less than one GPU-day through parameter sharing.
- DARTS introduced differentiable NAS for improved performance and efficiency, followed by SNAS for one-hot architecture parameters and ProxylessNAS for target-task search.
3. Method
P-DARTS extends DARTS by progressively increasing search depth while reducing the operation space and regularizing skip-connect bias. It uses staged search to narrow the gap between shallow search and deep evaluation.
- DARTS baseline: P-DARTS represents each cell as a DAG whose edges combine candidate operations weighted by learned architecture parameters.Intermediate nodes aggregate incoming edge outputs, and the cell output concatenates intermediate-node signals.
- Progressively increasing depth: DARTS searches with 8 cells but evaluates with 20, creating a depth gap because shallow-search preferences may differ from deep-evaluation preferences.The paper reports that discovered normal cells tend to retain shallow connections, partly because shallow networks descend faster during search.
- Search space approximation: P-DARTS increases network depth across stages while reducing candidate operations to control the exponential time and memory growth of deeper search.Lower-weight operations from the previous stage are dropped, making the procedure deployable on regular GPUs such as those with 16GB memory.
- Search space regularization: Search space regularization applies operation-level Dropout to skip-connect paths because deeper differentiable searches tend to favor their fast optimization despite weaker representation learning.The second regularization technique must be applied on top of operation-level Dropout to produce useful architecture weights.
- Staged search: The method restarts network-weight training after each stage and determines the final cell topology from final architecture scores, retaining the top-weighted nonzero operations.Warm-starting from the previous stage performed worse because its prior favored architectures suitable for shallow networks.
- Relationship to prior work: P-DARTS differs from PNAS by progressively searching at the cell level to enlarge architecture depth, whereas PNAS searches operations within cells to reduce evaluations.The paper also distinguishes its depth-based approach from SNAS’s one-hot architecture-weight strategy.
4.1. Datasets
The experiments search architectures on CIFAR10 and CIFAR100, evaluate them on those datasets, and test transferability to ImageNet’s ILSVRC2012 benchmark.
- Datasets: Experiments use CIFAR10, CIFAR100, and ImageNet, with architecture search conducted on CIFAR10 and CIFAR100 and evaluation conducted on all three datasets.CIFAR10 and CIFAR100 each contain 50K training and 10K testing RGB images at 32×32 resolution.
- Datasets: ILSVRC2012 tests transferability and contains 1,000 categories with 1.28M training and 50K validation images.The ImageNet experiments use 224×224 inputs and restrict multi-add operations to below 600M in the mobile setting.
4.2. Architecture Search
Architecture search uses three progressively deeper stages with regularization and constrained skip-connects. P-DARTS supports CIFAR100 search and reports strong CIFAR10 results at substantially lower search cost than prior methods.
- Architecture Search: The search network grows from 5 to 11 to 17 cells across three stages, with stage-specific skip-connect Dropout probabilities.For CIFAR10, the initial probabilities are 0.0, 0.4, and 0.7 across stages; for CIFAR100, they are 0.1, 0.2, and 0.3.
- Architecture Search: P-DARTS architectures preserve more deep connections and produce cells with more cascading levels than DARTS.The paper associates these deeper evaluation networks with better classification performance.
- Search Results: P-DARTS enables architecture search on CIFAR100, where prior approaches mostly failed, and its discovered architecture outperforms transferred architectures.A DARTS CIFAR100 search produced a skip-connect-heavy architecture with much worse classification performance.
4.3. Architecture Evaluation
P-DARTS achieves strong classification and transfer performance across CIFAR10, CIFAR100, and ImageNet evaluations, while using substantially less search computation than prior approaches.
- P-DARTS outperforms ENAS in classification performance and search cost while using fewer parameters.
- Architecture transfer between CIFAR10 and CIFAR100 shows dataset-specific bias in NAS evaluation.Architectures discovered on each dataset perform better when evaluated on that same dataset.
- P-DARTS architectures discovered on CIFAR10 and CIFAR100 outperform DARTS on ImageNet classification performance.The ImageNet evaluation uses architectures transferred from both proxy datasets.
- P-DARTS achieves lower ImageNet test error than MnasNet and ProxylessNAS, despite their ImageNet-specific search spaces.
4.4. Diagnostic Experiments
Diagnostic experiments show that deeper progressive search, search-space approximation, and regularization each contribute to the discovered architecture's performance, efficiency, or stability.
- The stage-3 architecture achieves the lowest test error among architectures extracted from the progressive search stages.
- Deeper search networks preserve more deep connections and cause discovered architectures to select intermediate nodes for later nodes.
- Approximately 7 hours of search on one Nvidia Tesla P100 GPU produces the final architectures, with stable memory usage and rare out-of-memory errors.
- 2.58% test error with an approximated search space beats 3.43% from a randomly sampled search space, supporting approximation's role in accuracy.
- Test error generally increases as preserved skip-connect operations rise from 0 to 4, while parameter count decreases from 4.1M to 2.7M.The reported exception is M = 0 for the test-error trend.
- Architecture refinement reduces variation from a pre-refinement test error of 2.79 ± 0.16%.Before refinement, evaluated architectures contain 2, 3, or 4 skip-connect operations in normal cells.
5. Conclusions
P-DARTS progressively increases search depth to bridge the search–evaluation depth gap, using approximation and regularization to manage the resulting costs and instability. The approach achieves fast CIFAR10 search and strong performance on both proxy and target datasets, while emphasizing depth as an important search factor.
- P-DARTS gradually increases candidate-architecture depth during search to bridge the depth gap between search and evaluation.
- Search-space approximation and regularization address the computational overhead and instability introduced by deeper progressive search.
- P-DARTS reaches a 3% CIFAR10 error rate with the fastest NAS speed reported in the paper and achieves superior proxy- and target-dataset performance.
- The paper argues that depth is important in differentiable architecture search and that approximation can trade width, measured by operation count, for depth.