PaperlayerView exact revision on arXiv

Source-linked AI summary

MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation

Chaoyang He, Haishan Ye, Li Shen, Tong Zhang

arXiv:2003.12238v1cs.LGcs.CVstat.ML

TL;DR

Bilevel NAS relies on approximations that can introduce gradient error, while single-level optimization can overfit architecture parameters. MiLeNAS mixes the two optimization schemes; experiments report better validation accuracy three times faster than bilevel optimization.

  • Problem

    Bilevel NAS is difficult to solve exactly because architecture updates require fully training network weights, and approximations can introduce gradient error; single-level optimization can overfit architecture parameters.

    Source: §1 · p. 1

  • Method

    MiLeNAS reformulates NAS as mixed-level optimization, jointly optimizing training and validation losses with respect to architecture parameters and training loss with respect to network weights.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    MiLeNAS achieves better validation accuracy three times faster than bilevel optimization; on CIFAR-10, its error rate is 2.51% ± 0.11% (best: 2.34%), versus DARTS-2.76% and GDAS-2.82%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework extends beyond DARTS to GDAS and sampling-based search, with model size-based search and early stopping accelerating search to 5 hours.

    Source: §1 · p. 2

Abstract

from arXiv · show

Many recently proposed methods for Neural Architecture Search (NAS) can be formulated as bilevel optimization. For efficient implementation, its solution requires approximations of second-order methods. In this paper, we demonstrate that gradient errors caused by such approximations lead to suboptimality, in the sense that the optimization procedure fails to converge to a (locally) optimal solution. To remedy this, this paper proposes \mldas, a mixed-level reformulation for NAS that can be optimized efficiently and reliably. It is shown that even when using a simple first-order method on the mixed-level formulation, \mldas\ can achieve a lower validation error for NAS problems. Consequently, architectures obtained by our method achieve consistently higher accuracies than those obtained from bilevel optimization. Moreover, \mldas\ proposes a framework beyond DARTS. It is upgraded via model size-based search and early stopping strategies to complete the search process in around 5 hours. Extensive experiments within the convolutional architecture search space validate the effectiveness of our approach.

1. Introduction

MiLeNAS addresses the tension between bilevel NAS, whose approximations can introduce gradient error, and single-level optimization, which can overfit architecture parameters. It mixes the two optimization schemes and reports better validation performance with faster search, alongside extensions to gradient-based NAS.

1. Introduction

  • Bilevel NAS is difficult to solve because each architecture update requires fully training the network, and first- or second-order approximations can leave the inner optimum unsatisfied and distort gradients.Second-order approximation builds on a one-step weight approximation, compounding gradient error.Source: §1 · p. 1
  • Single-level optimization can overfit architecture parameters because they depend only on training data, leaving validation loss insufficiently controlled.The framework also adds model size-based search and early stopping to accelerate the search process.Source: §1 · pp. 1–2
  • MiLeNAS simultaneously minimizes training loss with respect to network weights and both training and validation losses with respect to architecture parameters, mixing single-level and bilevel optimization.At λ = 0 the method reduces to single-level optimization, while as λ approaches infinity it becomes bilevel optimization.Source: §1 · pp. 1–2
  • MiLeNAS is presented as a framework beyond DARTS, with experiments in DARTS and GDAS settings and application to sampling-based search.The model size-based and early stopping strategies accelerate search to 5 hours.Source: §1 · p. 2
  • Three times faster than bilevel optimization, MiLeNAS achieves better validation accuracy; its CIFAR-10 error rate is 2.51% ± 0.11% (best: 2.34%).This exceeds DARTS-2.76% and GDAS-2.82%.Source: §1 · p. 2

2. Related Works

Related NAS work includes evolutionary, reinforcement-learning, and gradient-based approaches, each with different search mechanisms and efficiency constraints. Other recent methods target distinct aspects of NAS, while some accuracy gains rely on new search spaces or search strategies rather than general optimization methods.

2. Related Works

  • Evolutionary NAS can optimize architectures and weights simultaneously but may require enormous compute, while gradient-based NAS relaxes categorical choices and can search within several GPU days.Reinforcement-learning NAS formulates network design as actions and model accuracy as a reward.Source: §2 · p. 2
  • Gradient-based NAS is the category this work seeks to improve in efficiency and effectiveness.Source: §2 · p. 2
  • Recent NAS work also addresses multi-task learning, hand-designed heuristics, and the gap between search and evaluation accuracy.The passage notes that newer methods achieve higher accuracy than MiLeNAS, attributing their improvements to novel search spaces or strategies rather than a fundamental generic optimization method.Source: §2 · p. 2

3. Proposed Method

MiLeNAS reformulates NAS as mixed-level optimization, combining training and validation losses to address architecture-parameter overfitting. The method is compared with DARTS variants and extended to additional search spaces, model-size-based search, and early stopping.

3.1. Mixed-Level Reformulation

  • MiLeNAS constrains validation loss relative to training loss, yielding a weighted objective that balances both losses and can alleviate architecture-parameter overfitting.The non-negative parameter λ balances the training and validation losses.Source: §3.1 · p. 3

3.2. Comparison between MiLeNAS and DARTS

  • DARTS-2nd's stacked approximations can introduce gradient error and require multiple forward and backward passes, while MiLeNAS-1st avoids that approximation and uses fewer operations.The authors associate the approximation error with unreliable search and potentially suboptimal architectures.Source: §3.2 · p. 4
  • MiLeNAS-1st uses training and validation losses to update architecture parameters, whereas DARTS-1st uses validation loss alone.The paper reports better performance for MiLeNAS-1st than DARTS-1st.Source: §3.2 · p. 4
  • MiLeNAS-2nd is reported to search more slowly and yield lower-accuracy architectures than MiLeNAS-1st, supporting the authors’ preference for MiLeNAS-1st.The paper attributes the difference to second-order approximation gradient error.Source: §3.2 · p. 4

3.3. Beyond the DARTS Framework

  • The framework applies mixed-level optimization in both DARTS’s mixed-operation space and GDAS’s differentiable sampling space.For GDAS, the paper substitutes mixed-level optimization for bilevel optimization.Source: §3.3 · pp. 4–5
  • Model size-based search tracks the best validation accuracy at each model size in one run, while early stopping halts when model size falls below an expected size.The stopping point is selected by analyzing the relationship between model size and performance.Source: §3.3 · p. 5

4. Experiments and Results

Across CIFAR-10 and ImageNet experiments, MiLeNAS showed favorable accuracy and search-efficiency comparisons with bilevel and other NAS methods. Its mixed-level settings also varied the training–validation accuracy gap between the single-level and bilevel cases.

4.1. Settings

  • Experiments used a DARTS-similar search space, four search runs per method, and the same hyperparameters as DARTS for fair comparison.Source: §4.1 · p. 5

4.2. Comparison with Single-level and Bilevel methods

  • The training–validation accuracy gap was largest for single-level and smallest for bilevel optimization, while MiLeNAS’s gap shifted between them with λ.The experiment used λ = 0.1, 1, and 999 for MiLeNAS; the authors relate the endpoints λ = 0 and λ → ∞ to single-level and bilevel optimization, respectively.Source: §4.2 · p. 6
  • MiLeNAS with λ = 1 achieved the highest validation accuracy; DARTS-2nd exceeded DARTS-1st, while single-level optimization had the lowest validation accuracy.Source: §4.2 · p. 6, Figure 2 · p. 6
  • After 35 epochs of bilevel optimization, switching to MiLeNAS continued improving late-search validation accuracy, and MiLeNAS was over three times faster than DARTS-2nd.Source: §4.2 · p. 6

4.3. Evaluation Results on CIFAR-10

  • MiLeNAS matched state-of-the-art RL- and evolution-based NAS test error on CIFAR-10 using three orders of magnitude fewer computation resources, and beat ENAS, DARTS-2nd, SNAS, and GDAS in error and parameter count.The evaluation stacked 20 searched cells and trained the network from scratch for 600 epochs.Source: §4.3 · p. 6, Table 1 · p. 7

4.4. Transferability on ImageNet

  • On ImageNet, CIFAR-10-searched cells transferred successfully, with MiLeNAS finding smaller cells at relatively better performance and search speeds three times faster than DARTS-2nd.Source: §4.4 · p. 6, Table 2 · p. 7

5. Beyond the DARTS Framework

MiLeNAS was applied beyond DARTS, including a GDAS sampling-based setting, and the paper introduces model-size tracking and early stopping strategies. The observed search pattern links model size to accuracy and motivates a shorter search procedure.

5. Beyond the DARTS Framework

  • Replacing GDAS’s bilevel optimization with MiLeNAS (Gumbel) raised validation accuracy from 65.79% to 69.56% and reduced error from 2.82% to 2.57%.Source: §5 · p. 7

5.1. Model Size-based Searching

  • Model-size-based searching tracks the best validation accuracy at different sizes, counting convolution operations because discrete cell operations determine model size.Source: §5.1 · pp. 7–8
  • MiLeNAS search proceeded in size-specific phases, reducing model size between phases; accuracy increased with size but plateaued in the optimal 3.5M–4.5M range.The authors report that DARTS did not consistently show the same decreasing-size pattern, requiring more search rounds to summarize its size–accuracy relationship.Source: §5.1 · p. 8, Figure 3 · p. 8
  • Model-size-based searching evaluates architectures at different sizes, potentially identifying economical or compression-oriented alternatives and a regular size–accuracy pattern for refining search.Source: §5.1 · p. 8

5.2. Early Stopping Strategy

  • Early stopping uses the early appearance of the optimal parameter range; stopping at six normal-cell and zero reduction-cell convolution operations cut CIFAR-10 search to around five hours.Source: §5.2 · p. 8

6. Conclusion

The conclusion presents MiLeNAS as a mixed-level reformulation intended to reduce bilevel approximation gradient error while retaining first-order efficiency. Experiments support faster search and lower validation error, with model-size search and early stopping offering further speedups.

  • MiLeNAS reformulates NAS as mixed-level rather than bilevel optimization, aiming to alleviate approximation gradient error while retaining first-order efficiency.Source: §6 · p. 8
  • Image-classification experiments showed lower validation error with search time three times shorter than second-order bilevel optimization.Source: §6 · p. 8
  • The conclusion describes MiLeNAS as generic, with sampling-based applicability experiments and model-size search plus early stopping providing further search speedups and design insights.Source: §6 · p. 8

A. Experiment Details

The experiment details specify the CIFAR-10 operation set and the cell structure used to define architecture parameters and network weights.

A.1. Search Space Definition

  • The CIFAR-10 search space uses eight operations: separable and dilated separable convolutions, max and average pooling, identity, and zero.Source: §A.1 · p. 11
  • Each cell has two input nodes, four intermediate nodes, and an output node concatenating intermediate outputs; reduction cells reduce feature-map resolution at one-third and two-thirds of network depth.Source: §A.1 · p. 11
  • Normal cells and reduction cells share separate architecture parameters, and the method alternately optimizes those parameters and network weights.Source: §A.1 · p. 11

A.2. Transferability on ImageNet

ImageNet transfer training uses a 14-cell network trained for 250 epochs on eight NVIDIA Tesla V100 GPUs.

A.2. Transferability on ImageNet

  • The model is restricted to less than 600M parameters.Source: §A.2 · p. 11
  • Training uses 14 cells, 250 epochs, and a batch size of 128.Source: §A.2 · p. 11
  • Training takes around three days on a server with eight NVIDIA Tesla V100 GPUs.Source: §A.2 · p. 11

A.3. Searched Architecture

The section points to Figure 5 for examples of the searched architectures.

A.3. Searched Architecture

  • Figure 5 shows examples of the searched architectures.Source: §A.3 · p. 11

B. Derivation of the MiLeNAS Second-Order Method

The second-order MiLeNAS method approximates the validation gradient using a one-step update to the weights, then simplifies its second-order derivative with finite differences.

B. Derivation of the MiLeNAS Second-Order Method

C. Evaluation on MiLeNAS-2nd

The evaluation reports that MiLeNAS-2nd has lower validation accuracy than MiLeNAS-1st and the longest search time; Figure 4 compares the two MiLeNAS variants.

C. Evaluation on MiLeNAS-2nd

  • MiLeNAS-2nd has lower validation accuracy than MiLeNAS-1st, similar accuracy to DARTS-2nd, and the longest search time.Each method was run four times, with results averaged.Source: C. Evaluation on MiLeNAS-2nd · p. 11, Figure 4 · p. 11
  • Figure 5 shows a searched architecture with a 2.34% error rate and 3.87M parameters, and another with a 2.50% error rate and 2.86M parameters.Source: Figure 5 · p. 12
Loading 2003.12238v1…