Balanced One-shot Neural Architecture Optimization

1 Introduction

• Training each candidate architecture until convergence makes NAS resource-intensive.NAS iteratively generates candidates for evaluation and learns from their results.Source: §1 · p. 1
• One-shot NAS reduces resource demands through a shared supernet, but its discovered architectures show inferior performance and weak stability compared with conventional NAS.The figure contrasts ranking preservation in conventional NAS with poor preservation in one-shot NAS.Source: §1 · p. 1, Figure 2 · p. 2
• Short one-shot training weakens ranking correlation because architectures receive insufficient optimization, and small models can fare better than large ones despite the reverse at full training.The validation accuracy is therefore less representative of ground-truth performance; the paper reports that ranking correlation increases with training time.Source: §1 · p. 2
• Balanced NAO samples architectures in proportion to model size, giving large architectures more supernet updates than small architectures.The expected training time of an architecture associates with its size, and experiments show that ranking correlation increases.Source: Figure 1 · p. 1, §1 · p. 2
• 2.60% test error on CIFAR-10 and 74.4% top-1 accuracy on ImageNet accompany significant improvements over baseline one-shot NAS methods.The proposed method also generates architectures with more stable performance and lower variance.Source: §1 · p. 2

2 Related Work

• Reinforcement learning, evolutionary algorithms, and gradient-based search have achieved promising results on several tasks, but these methods are costly.Examples include Zoph and Le (2016), Real et al. (2018), and Luo et al. (2018).Source: §2 · p. 2
• One-shot NAS reduces conventional NAS resource demands by sharing candidate weights in a supernet; ENAS uses RL, DARTS bi-level gradient optimization, and NAO weight sharing.These approaches are described by Bender et al. (2018), Pham et al. (2018), Liu et al. (2018b), and Luo et al. (2018).Source: §2 · p. 2
• ProxylessNAS binaries architecture-associated parameters to reduce DARTS memory and search large-scale tasks directly, while SNAS jointly trains supernet and architecture-distribution parameters.The cited works are Cai et al. (2018) and Xie et al. (2018).Source: §2 · pp. 2–3
• Supernet-based one-shot approaches require delicate design and careful tuning.Source: §2 · p. 3

3.1 Stability of Baselines

• In five runs, ENAS, DARTS, and NAO-WS each sometimes matched their reported CIFAR-10 error, but results were unstable and all trailed conventional NAS.The conventional NAS examples reported were AmoebaNet-B at 2.55% and NAONet at 2.48%.Source: §3.1 · p. 3, Table 1 · p. 3

3.2 Insufficient Optimization

• 62% pairwise accuracy under short one-shot training was only slightly better than random, indicating that architecture rankings were poorly preserved.Pairwise accuracy increased as training time and the average updates per architecture increased; practical searches sample thousands of architectures, leaving each with only a handful of updates.Source: §3.2 · pp. 3–4, Table 2 · p. 3

3.3 Imbalanced Training of Architectures

• Pairwise accuracy was lower for diverse-size architectures than similar-size ones; small models could look better during one-shot training yet generally fare worse after full training.The authors attribute this pattern to small architectures being easier to optimize with limited updates, while larger architectures receive relatively less training.Source: §3.3 · p. 4, Table 3 · p. 4

4 Method

• NAO is chosen because it samples from an architecture pool, whereas DARTS-like methods combine parameter and architecture optimization in a forward pass.The paper notes that this makes NAO’s sampling procedure easier to control.Source: §4 · p. 4
• Allocating training steps directly in proportion to model size may exceed the search budget, and training time need not scale linearly with size.The paper therefore uses proportional sampling to balance architectures during supernet training.Source: §4 · pp. 4–5
• Sampling architectures in proportion to model size makes expected training steps proportional to each architecture’s size.At each supernet training step, an architecture is sampled from the candidate pool; over T_t total steps, its expected updates scale with its model size.Source: §4 · p. 5
• Balanced NAO incorporates proportional architecture sampling into one-shot NAO with weight sharing.The method is based on NAO because it samples from an architecture pool during supernet training.Source: §4 · pp. 4–5

5.1 Ranking Correlation Improvement

• More than 10 points higher pairwise accuracy with proportional sampling indicates improved agreement between one-shot and ground-truth rankings.Source: Table 6 · p. 5, §5.1 · pp. 5–6

5.2 Stability Study

• 2.67% average and 2.60% best CIFAR-10 test error across five architectures; all five beat baselines and had lower variance.Source: Table 7 · p. 6, §5.2 · p. 6

5.3 Compared to Various NAS Methods

• Balanced NAO outperformed baseline one-shot NAS and was on par with conventional NAS under similar model size, despite the latter's higher search cost.Conventional methods may require hundreds or thousands of GPU-days, while mainstream one-shot methods use fewer than ten.Source: §5.3 · p. 6

5.4 Transfer to Large Scale Imagenet

• 74.4% top-1 accuracy on ImageNet surpassed listed human-designed models; Balanced NAO was slightly worse than methods searching ImageNet directly with more careful designs and more complex search spaces.Source: §5.4 · p. 6

• Insufficient optimization in one-shot NAS leaves architectures of different sizes imbalanced, introducing incorrect rewards and weakening ranking correlation.Source: §6 · p. 6
• Balanced NAO addresses this imbalance by balancing architectures through proportional sampling.Source: §6 · p. 6
• Experiments show improvements in performance and stability; further investigation of one-shot NAS is proposed as future work.Source: §6 · p. 6