Source-linked AI summary
BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, Quoc Le
TL;DR
Weight-sharing NAS often needs retraining or post-processing because shared-weight accuracies lag stand-alone training. BigNAS trains one broad single-stage model and selects directly deployable child models, whose ImageNet results surpass state-of-the-art models from 200 MFLOPs to 1 GFLOPs.
Problem
Shared-weight NAS can rank architectures, but its absolute accuracies are typically below stand-alone training, commonly requiring costly retraining or post-processing.
Method
BigNAS simultaneously trains child models in one single-stage model and uses coarse-to-fine selection to slice models for direct deployment.
Results
BigNAS slices surpass all state-of-the-art ImageNet classification models ranging from 200 MFLOPs to 1 GFLOPs.
Takeaways & Limitations
A single pretrained shared-weight model can provide high-quality child models of different sizes for instant deployment without retraining or finetuning.
Abstract
from arXiv · showhide
Neural architecture search (NAS) has shown promising results discovering models that are both accurate and fast. For NAS, training a one-shot model has become a popular strategy to rank the relative quality of different architectures (child models) using a single set of shared weights. However, while one-shot model weights can effectively rank different network architectures, the absolute accuracies from these shared weights are typically far below those obtained from stand-alone training. To compensate, existing methods assume that the weights must be retrained, finetuned, or otherwise post-processed after the search is completed. These steps significantly increase the compute requirements and complexity of the architecture search and model deployment. In this work, we propose BigNAS, an approach that challenges the conventional wisdom that post-processing of the weights is necessary to get good prediction accuracies. Without extra retraining or post-processing steps, we are able to train a single set of shared weights on ImageNet and use these weights to obtain child models whose sizes range from 200 to 1000 MFLOPs. Our discovered model family, BigNASModels, achieve top-1 accuracies ranging from 76.5% to 80.9%, surpassing state-of-the-art models in this range including EfficientNets and Once-for-All networks without extra retraining or post-processing. We present ablative study and analysis to further understand the proposed BigNASModels.
1 Introduction
BigNAS targets the deployment gap in weight-sharing NAS: shared weights can rank architectures but usually require retraining or post-processing for competitive absolute accuracy. It trains one broad single-stage model whose child models can be directly deployed across resource constraints.
- Device and driver differences can make architectures with similar overall speed favor different deployment outcomes.
- Weight-sharing NAS reduces search cost by training a super-network and selecting an efficient path under hardware constraints.
- Shared-weight predictions typically underperform stand-alone models, motivating retraining or post-processing before deployment.
- BigNAS trains a single-stage model that directly yields high-quality child models without extra post-processing.
- BigNAS spans approximately 200 MFLOPs to 1 GFLOPs and supports instant deployment by slicing the trained model under memory or latency constraints.Architecture selection uses a coarse-to-fine strategy after simultaneous child-model training.
- Training the shared model is challenging because big child models can overfit while small child models have not yet converged.The paper addresses distinct initialization and learning dynamics across child-model sizes.
2 Related Work
Related NAS work progressed from expensive stand-alone candidate training to weight-sharing models and resource-aware search, but generally retained retraining or narrow budget coverage. BigNAS instead trains all child models simultaneously across a wider architectural space.
- Earlier NAS methods trained thousands of candidate architectures from scratch, whereas one-shot methods rank child architectures using shared weights.
- Resource-aware NAS jointly considers prediction accuracy and requirements such as latency, FLOPs, or memory footprint.
- Existing approaches generally require two-stage training and often target one resource budget or a narrow budget range per search.
- OFA progressively distills a pretrained full network and uses channel sorting to construct smaller networks.
- BigNAS trains all child models simultaneously and searches multiple dimensions, including kernel size, channels, depth, and input resolution.
3 Architecture Search with Single-Stage Models
BigNAS trains and selects architectures within one shared single-stage model, addressing the differing optimization behavior of small and big children. Its procedure combines specialized training techniques with coarse-to-fine resource-aware search and direct deployment.
- Method overview: BigNAS trains one large single-stage model from which child architectures are sliced for deployment without retraining or post-processing.
- Method overview: The training study covers network sampling, inplace distillation, initialization, convergence behavior, and regularization before architecture selection.
- Training a High-Quality Single-Stage Model: The sandwich rule samples the smallest, largest, and N random child models, then aggregates their gradients to update shared weights.In the experiments, N = 2; the smallest model varies resolution, width, depth, and kernel size together.
- Training a High-Quality Single-Stage Model: Inplace distillation uses the full model’s soft labels to supervise other child models during each shared-weight update.
- Training a High-Quality Single-Stage Model: Shared training is unstable because larger models can overfit sooner while smaller models converge more slowly.Standard learning rates caused exploding loss, while reducing the rate to 30% produced an approximately 1.0% top-1 accuracy drop on ImageNet.
- Training a High-Quality Single-Stage Model: An exponentially decaying learning rate with constant ending helps small models learn faster and alleviates big-model overfitting through weight oscillation.
- Coarse-to-fine Architecture Selection.: Coarse-to-fine selection benchmarks a restricted space, chooses a promising skeleton under the resource constraint, and mutates its architectural dimensions for refinement.The resulting child weights are used directly without retraining or finetuning.
4 Experiments
BigNAS evaluates a broad MobileNetV2-based search space and trains one shared single-stage model to produce accurate child models across resource budgets. ImageNet benchmarks and ablations examine performance, initialization, convergence, regularization, and coarse-to-fine architecture selection.
- Main Results on ImageNet: 76.5% top-1 accuracy is achieved by BigNASModel-S at 240 MFLOPs, exceeding MobileNetV3 by 1.3% at similar FLOPs.BigNASModel-L also improves on EfficientNet B2 by 0.6%, while selected models are sliced directly from one pretrained model.
- Ablation Study: 0% validation accuracy occurs for both small and big child models after a few thousand steps when He Initialization is used during warm-up.The ablation motivates initialization designed for the varying number of input units across child models.
- Ablation Study: Big child models converge faster and overfit, whereas small child models converge more slowly and require more training.The proposed exponentially decaying schedule with a constant ending learning rate addresses these different convergence behaviors.
- Ablation Study: +0.5 top-1 accuracy is obtained for small child models, while big child models improve by +0.2 when regularization is applied only to the full network.The comparison uses weight decay of 10^-5 and dropout ratio 0.2.
- Architecture Selection: Coarse-to-fine selection first benchmarks rough architecture skeletons, then randomly varies resolutions, depths, channels, and kernel sizes near a selected candidate.The approach evaluates architectures under flexible resource metrics including accuracy, FLOPs, memory footprint, and device latency.
5 Analysis of BigNASModel
The analysis tests whether BigNAS child models benefit from finetuning or independent training. Results indicate that finetuning adds little, while weight-sharing distillation provides a slight benefit when training child architectures from scratch.
- Finetuning Child Models: Finetuning BigNASModel-S, -M, -L, and -XL for 25 additional epochs no longer improves accuracy significantly.The experiments use different constant learning rates for each child model.
- Training from Scratch: Child architectures trained from scratch slightly benefit from jointly training a weight-sharing single-stage model.This result is consistent with observations from previous work.
- Training from Scratch: Simple distillation without temperature does not help EfficientNet-based architectures in the reported experiments.The second distillation variant is inplace distillation, which is most similar to the distillation used for BigNASModel training.
6 Conclusion
BigNAS introduces a single-stage NAS paradigm that directly induces high-quality child models of different sizes from one pretrained model. The resulting BigNASModels surpass reported ImageNet state-of-the-art models from 200 MFLOPs to 1 GFLOPs without retraining or finetuning.
- Conclusion: BigNASModels are induced as high-quality child models of different sizes from one pretrained single-stage model.The models can be deployed without retraining or finetuning.
- Conclusion: BigNASModels simultaneously surpass state-of-the-art ImageNet classification models ranging from 200 MFLOPs to 1 GFLOPs.The conclusion presents this as evidence for simplifying and scaling up neural architecture search.
BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models (Supplementary Materials)
The supplementary-materials passage supplied here contains author and affiliation information only, with no substantive experimental or methodological content.
- Supplementary Materials: The listed authors include Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, and Quoc Le.The passage also lists Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, and Ruoming Pang.
A Architectures of BigNASModel
BigNASModel-XL is directly sliced from the single-stage model without retraining or finetuning, achieving higher top-1 accuracy than EfficientNet-B2 at a similar computational budget.
- 80.9% top-1 accuracy under 1040 MFLOPs is achieved by BigNASModel-XL, compared with 80.3% under 1050 MFLOPs for EfficientNet-B2.
B Learning Rate Schedule: Exponentially Decaying with Constant Ending
BigNAS uses a constant ending learning rate to address different convergence behaviors among small and big child models, with the minimum set to 5% of the initial rate.
- 5% of the initial learning rate is used as the minimum in the constant ending learning-rate schedule.The value was intended as a small constant rather than specifically tuned, and results were insensitive to this hyper-parameter.
C Implementation Details
BigNAS implements training and architecture selection on TensorFlow using TPUv3 pods, with static-graph masking and synchronized batch normalization to support directly sliced child models.
- Training and coarse-to-fine architecture selection are implemented in TensorFlow on 8×8 TPUv3 pods.
- Average Top-1 accuracy remains similar when the minimum learning rate is varied among 3%, 5%, 8%, and 10%.
- The ImageNet experiments use a total batch size of 4096, while the single-stage model spans 200–2000 MFLOPs and searched architectures span 200–1000 MFLOPs.Training a single-stage model roughly takes 36 hours.
- Static TPU graphs are supported by masking kernel sizes, channels, and depths to simulate weight slicing and path selection.
- Synchronized batch normalization is used across TPU replicas during training.Stateless random sampling ensures consistent sampled values across TPU cores and supports different layer-wise and stage-wise configurations.
- Figure 1 depicts the single-stage model and BigNASModel-S, -M, -L, and -XL child models, which are directly sliced without retraining or finetuning.