PaperlayerView exact revision on arXiv

Source-linked AI summary

DSNAS: Direct Neural Architecture Search without Parameter Retraining

Shoukang Hu, Sirui Xie, Hehui Zheng, Chunxiao Liu, Jianping Shi, Xunying Liu, Dahua Lin

arXiv:2002.09128v2cs.LGstat.ML

TL;DR

Two-stage NAS often evaluates architectures after retraining even though search-stage and retrained performance correlate poorly. DSNAS defines task-specific end-to-end NAS and jointly optimizes architecture and parameters, producing deployable subnetworks without retraining. It reports 74.4% performance on ImageNet classification in 420 GPU hours, with total time reduced by 34% versus two-stage NAS.

  • Problem

    Two-stage NAS commonly evaluates retrained architectures despite low correlation between search-stage and retrained performance under parameter sharing.

    Source: §1 · p. 1

  • Method

    DSNAS is a differentiable NAS framework that jointly optimizes architecture and parameters in one back-propagation round to produce subnetworks deployable without retraining.

    Source: §1 · p. 2

  • Results

    74.4% performance on ImageNet classification was achieved in 420 GPU hours, reducing total time by 34% compared with two-stage NAS.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Task-specific end-to-end NAS evaluates search efficiency and final task performance, with the searched network intended for that task rather than other tasks.

    Source: §1 · p. 2

    SNAS's Gumbel-softmax relaxation requires storing the whole parent network in GPU memory, preventing direct application to large-scale networks.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

If NAS methods are solutions, what is the problem? Most existing NAS methods require two-stage parameter optimization. However, performance of the same architecture in the two stages correlates poorly. In this work, we propose a new problem definition for NAS, task-specific end-to-end, based on this observation. We argue that given a computer vision task for which a NAS method is expected, this definition can reduce the vaguely-defined NAS evaluation to i) accuracy of this task and ii) the total computation consumed to finally obtain a model with satisfying accuracy. Seeing that most existing methods do not solve this problem directly, we propose DSNAS, an efficient differentiable NAS framework that simultaneously optimizes architecture and parameters with a low-biased Monte Carlo estimate. Child networks derived from DSNAS can be deployed directly without parameter retraining. Comparing with two-stage methods, DSNAS successfully discovers networks with comparable accuracy (74.4%) on ImageNet in 420 GPU hours, reducing the total time by more than 34%. Our implementation is available at https://github.com/SNAS-Series/SNAS-Series.

1. Introduction

Two-stage NAS can rank architectures poorly because search and retraining use different parameter-training schemes, yet retraining is commonly used to evaluate candidates. DSNAS instead defines task-specific end-to-end NAS and jointly optimizes architecture and parameters for a deployable network without retraining.

1. Introduction

  • Search and retraining performance correlate poorly when parameter sharing is used, challenging the assumption that search-stage performance predicts retrained performance.The cited sources for this low correlation are [20] [5].Source: §1 · pp. 1–2
  • Task-specific end-to-end NAS evaluates a framework by search efficiency and final task performance, with a single training stage producing a ready-to-deploy network for that task.The searched network is not asserted to generalize to other tasks; the framework is the expected object of generalization.Source: §1 · p. 2
  • Two-stage methods incur extra computation from retraining and reproducibility, while SNAS's Gumbel-softmax relaxation stores the whole parent network and wastes memory and computation.One-shot methods can use single-stage training, but their prior sampling distribution can affect resources needed to reach satisfying accuracy.Source: §1 · p. 2
  • 74.4% performance on ImageNet classification was achieved by DSNAS in 420 GPU hours, reducing total time by 34% compared with two-stage NAS.The derived subnetworks are ready to deploy without parameter retraining.Source: §1 · p. 2
  • DSNAS jointly optimizes architecture and parameters in the same back-propagation round using a low-bias search gradient, and its subnetworks need no retraining.The gradient combines differentiable NAS stability and robustness with discrete-sampling NAS memory efficiency.Source: §1 · p. 2

2. Problem definition of NAS

Two-stage NAS separates architecture search from parameter retraining, while the relationship between search and retraining performance is not established. The section defines task-specific end-to-end NAS around a ready-to-deploy network and evaluates it by task accuracy and total computation.

2.1. Two-Stage NAS

  • Two-stage methods train parameters and optimize architecture during search, then commonly perform another round of parameter optimization for evaluation.The ideal of training every possible architecture from scratch is infeasible because of combinatorial complexity.Source: §2.1 · p. 3
  • Existing two-stage NAS leaves the objective vague, and reported work has not established correlation between search accuracy and retraining accuracy.Retraining is presented as an ad hoc response when search-stage accuracy is unsatisfying.Source: §2.1 · p. 3

2.2. Task-specific end-to-end NAS

  • Task-specific end-to-end NAS seeks a ready-to-deploy network with optimized architecture and parameters for a defined task.Source: §2.2 · p. 3
  • A task is bounded by its data and learning objective, so changing either factor changes what the searched result can be expected to achieve.The paper says this task-specific boundary can bring operational benefits to industrial applications.Source: §2.2 · p. 3
  • Evaluation uses final task accuracy and total time from taking in data to outputting a network with optimized architecture and parameters.For two-stage methods, the efficiency measure includes both search and evaluation time.Source: §2.2 · p. 3

3. Direct NAS without retraining

The section traces DSNAS from SNAS’s stochastic task-objective optimization, replacing continuous relaxation with a discrete policy-gradient estimate. It then describes sampled-subnetwork implementation, complexity considerations, progressive early stopping, and differences from one-shot NAS.

3.1. Stochastic Neural Architecture Search (SNAS)

  • SNAS models network construction as a stochastic computational graph, sampling edge operations to estimate expected task loss over architectures.Its Markov-chain reformulation represents feature maps as DAG nodes and candidate operations on edges.Source: §3.1 · pp. 3–4

3.2. Discrete SNAS (DSNAS)

  • DSNAS takes the temperature of SNAS’s policy-gradient-equivalent search gradient to zero, yielding a strictly one-hot architecture sample for the expected gradient.The method exploits the fact that policy gradients do not require the continuous relaxation used by the reparameterization trick.Source: §3.2 · p. 4

3.3. Implementation

  • DSNAS backpropagates through only the sampled subnetwork, using a dummy multiplier after selected operations to calculate the policy-gradient cost.This avoids instantiating the whole parent network with the batch dimension for backpropagation.Source: §3.3 · pp. 4–5, Figure 2 · p. 5

3.4. Complexity analysis

  • SNAS instantiates the whole graph, requiring n times the GPU memory and calculation of a sampled subnetwork; ProxylessNAS uses path sampling to reduce paths from n to 2.The ProxylessNAS approximation still theoretically involves the whole network, according to the section’s complexity discussion.Source: §3.4 · p. 5, Table 1 · p. 5

3.5. Progressive early stop

  • DSNAS progressively stops sampling and architecture updates on an edge once its stopping condition is satisfied, selecting that edge’s highest-probability operation.The strategy addresses non-zero architecture-distribution entropy that can sustain sampling and may hinder learning on other edges.Source: §3.5 · p. 6

3.6. Comparison with one-shot NAS

  • Unlike one-shot NAS methods described here, DSNAS jointly optimizes architecture and parameters rather than relying on pretraining or an auxiliary hypernetwork before architecture optimization.The section says one-shot approaches may take more epochs to reach comparable performance when investing more resources in pretraining; DSNAS is expected to save some pretraining resources.Source: §3.6 · p. 6

4. Experimental Results

The experiments test whether search-stage performance predicts evaluation-stage performance, then compare DSNAS with two-stage NAS on end-to-end accuracy and computation. They also break down computation time for DSNAS against differentiable NAS methods.

4.1. Accuracy correlation of two-stage NAS

  • Search-stage and evaluation-stage performance correlations are fairly low for SPOS and ProxylessNAS, motivating task-specific end-to-end NAS.The study measures Kendall Tau both across top-k models within one search and across top-1 models from repeated searches; τ_inter and τ_intra denote these two views.Source: §4.1 · p. 6, Table 2 · p. 6

4.2. Single-path architecture search

  • DSNAS is compared with SPOS configurations on ImageNet classification in a mobile setting, with a FLOPS constraint of 330M or less.The search space contains 420 single-path models, with four candidates per choice block; DSNASsearch240 and SPOSsearch240 use one-stage training for 240 epochs.Source: §4.2 · pp. 6–7, Figure 3 · p. 7
  • DSNAS achieves comparable accuracy with roughly 34% less computation than two-stage SPOS; SPOS models are around 1.4% and 4% less accurate, respectively, at epoch 240.Across the reported choice-block results, DSNAS uses at least one-third less computation, and its accuracy standard deviation is 0.22 versus 0.38 and 0.36 for SPOS’s search and evaluation stages.Source: §4.2 · p. 7, Table 3 · p. 8

4.3. Time consumption breakup

  • On CIFAR-10, DSNAS is almost five times faster than SNAS and uses only 1/n of its GPU memory, where n is the number of candidate operations per edge.The computation-time breakup covers forward, backward, optimization, and test; the comparison uses the same setting.Source: §4.3 · p. 8, Table 4 · p. 8
  • For ImageNet, the study compares DSNAS and ProxylessNAS computation times on the same search space with a total batch size of 512.The reported timing uses eight TITAN X GPUs for ProxylessNAS and four for DSNAS.Source: §4.3 · p. 8, Table 5 · p. 8

5. Summary and future work

DSNAS defines task-specific end-to-end NAS and jointly optimizes architecture and parameters. Its subnetworks are ready to deploy, and the framework is presented as complementary to topology search.

5. Summary and future work

  • DSNAS optimizes architecture and parameters in the same back-propagation round, producing subnetworks ready for deployment.The paper contrasts this with two-stage methods and notes that EfficientNet uses extra grid search on network scale.Source: §5 · p. 8
  • Searching network scales similarly to EfficientNet could yield more accuracy for DSNAS, while its total cost is reported as lower.Source: §5 · p. 8
  • DSNAS is orthogonal to random wiring, which searches graph topology; the authors propose combining them to search topology, operations, and parameters.Source: §5 · p. 8

A. Detailed Settings of Experimental Results

The experiments use standard image preprocessing with random crops or horizontal flips and per-channel pixel normalization.

A. Detailed Settings of Experimental Results

B. Details about the architectures

The architecture appendix presents choice blocks, a supernet layer-sequence table, and building-block search results for SPOS and a one-stage DSNAS search.

B. Details about the architectures

  • The choice-block figures show Choice 3, Choice 5, Choice 7, and Choice x, with separate illustrations for stride 1 and stride 2.Source: Figure 4 · p. 10, Figure 5 · p. 10
  • The appendix notes that its choice-block setting follows the released SPOS implementation [7].Source: B. Details about the architectures · p. 10
  • The supernet table describes repeated layer sequences, their output channels, and the stride of each sequence’s first layer.Later layers in each sequence use stride 1; CB, GAP, and FC denote choice block, global average pooling, and fully connected layer.Source: Table 6 · p. 10
  • One figure presents the implementation’s building-block search result in SPOS [7].Source: Figure 6 · p. 10
  • Another figure presents the Section 4 building-block search result from the one-stage DSANSsearch240 process.Source: Figure 7 · p. 10
Loading 2002.09128v2…