PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Architecture Transfer

Zhichao Lu, Gautam Sreekumar, Erik Goodman, Wolfgang Banzhaf, Kalyanmoy Deb, Vishnu Naresh Boddeti

arXiv:2005.05859v2cs.CVcs.LGcs.NE

TL;DR

Existing NAS requires costly searches for new tasks and deployment objectives, limiting its practicality across many scenarios. NAT adapts a pre-trained supernet while performing many-objective evolutionary search, producing task-specific subnetworks without additional training after adaptation. Across eleven image-classification tasks, NAT improves mobile performance and is reported as orders of magnitude more efficient than existing NAS methods.

  • Problem

    Existing NAS approaches have not efficiently produced both topology and weights for custom non-standard datasets because new tasks require large data and computational resources.

  • Method

    NAT alternates online supernet adaptation with many-objective evolutionary search and uses an online accuracy predictor to evaluate sampled subnets efficiently.

  • Results

    Across eleven benchmark image-classification datasets, NATNets outperform conventional fine-tuning under mobile settings, including 80.5% Top-1 accuracy on ImageNet at 600M MAdds.

  • Takeaways & Limitations

    NAT provides an effective alternative to conventional fine-tuning for diverse image-classification tasks and computational objectives, with especially strong benefits on small-scale fine-grained datasets.

  • Takeaways & Limitations

    The experiments manually limit computation to a maximum of 1 day on an 8-GPU NVIDIA 2080Ti server, representing small-lab resources.

Abstract

from arXiv · show

Neural architecture search (NAS) has emerged as a promising avenue for automatically designing task-specific neural networks. Existing NAS approaches require one complete search for each deployment specification of hardware or objective. This is a computationally impractical endeavor given the potentially large number of application scenarios. In this paper, we propose Neural Architecture Transfer (NAT) to overcome this limitation. NAT is designed to efficiently generate task-specific custom models that are competitive under multiple conflicting objectives. To realize this goal we learn task-specific supernets from which specialized subnets can be sampled without any additional training. The key to our approach is an integrated online transfer learning and many-objective evolutionary search procedure. A pre-trained supernet is iteratively adapted while simultaneously searching for task-specific subnets. We demonstrate the efficacy of NAT on 11 benchmark image classification tasks ranging from large-scale multi-class to small-scale fine-grained datasets. In all cases, including ImageNet, NATNets improve upon the state-of-the-art under mobile settings ($\leq$ 600M Multiply-Adds). Surprisingly, small-scale fine-grained datasets benefit the most from NAT. At the same time, the architecture search and transfer is orders of magnitude more efficient than existing NAS methods. Overall, the experimental evaluation indicates that, across diverse image classification tasks and computational objectives, NAT is an appreciably more effective alternative to conventional transfer learning of fine-tuning weights of an existing network architecture learned on standard datasets. Code is available at https://github.com/human-analysis/neural-architecture-transfer

1 INTRODUCTION

NAT addresses the computational burden of applying NAS to new tasks by adapting a pre-trained supernet while searching for task-specific architectures across objective trade-offs. It produces competitive custom models across diverse datasets and settings, including mobile image classification and dense prediction.

  • Motivation: Existing NAS methods typically require substantial data and computation to search for new architectures on each custom task.Earlier NAS methods perform well on standard benchmarks but have not broadly delivered both topology and weights for non-standard datasets.
  • Neural Architecture Transfer: NAT generates task-specific networks spanning multiple conflicting objectives without running NAS from scratch for every deployment specification.A single NAT run can produce multiple custom networks across the objective trade-off front.
  • Neural Architecture Transfer: NAT alternates supernet adaptation with many-objective evolutionary search to resolve the unknown relationship between promising subnets and the adapted supernet.The adaptation stage fine-tunes subnets sampled from a distribution built from promising evolutionary-search results.
  • Neural Architecture Transfer: An online accuracy predictor estimates subnet objectives without full evaluation, improving the computational efficiency of evolutionary search.The predictor is learned online from previously evaluated subnets.
  • Results: 80.5% Top-1 accuracy is achieved by NATNet on ImageNet at 600M MAdds, while NATNets lead to state-of-the-art performance across the evaluated mobile image-classification tasks.The evaluation spans large-scale, medium-scale, and small-scale multi-class and fine-grained datasets.
  • Results: NAT matches Auto-DeepLab’s mIoU on Cityscapes using 4× fewer MAdds and produces a best-trade-off model that dominates competing models across eleven datasets under mobile settings.The method also considers optimizing accuracy, model size, and MAdds, CPU latency, or GPU latency.

2 RELATED WORK

Related NAS work has pursued accuracy, transfer across tasks, efficient performance prediction, weight sharing, and multi-objective optimization. NAT combines these directions by adapting a reusable supernet to new tasks while coupling search with online prediction.

  • NAS Development: Early NAS methods used reinforcement learning or evolutionary algorithms to search modular architectures, often transferring architectures found on smaller datasets.The first wave primarily targeted predictive accuracy and commonly searched on CIFAR-10 or CIFAR-100 before transferring architectures.
  • NAS Transfer: Meta-learning, XferNAS, and EAT-NAS explored transferring architectures or search strategies across related datasets and task scales.Other approaches searched multiple tasks during training and transferred the learned search strategy to new tasks at inference.
  • Performance Prediction: Performance-prediction methods reduce evaluation cost by learning regressors that estimate architecture performance without fully training each architecture.Examples include radial-basis-function models, multilayer perceptrons, and recurrent neural networks.
  • Weight Sharing: Weight-sharing methods train supernets containing searchable architectures, but training a supernet from scratch for every new task remains computationally prohibitive.NAT instead trains a supernet on large-scale datasets once and couples it with search to adapt to new tasks.
  • NAT Positioning: NAT’s related-work position is to integrate supernet weight sharing, online performance prediction, and many-objective search for task-specific adaptation.Table 1 compares NAT with existing NAS methods and marks methods that scalarize multiple objectives.
  • Multi-Objective NAS: Multi-objective NAS methods either scalarize objectives or use Pareto-based solutions to balance accuracy with computational constraints.Scalarized approaches include ProxylessNAS, MnasNet, ChamNet, MobileNetV3, and FBNetV2.

3 PROPOSED APPROACH

Neural Architecture Transfer combines an accuracy predictor, many-objective evolutionary search, and supernet adaptation to efficiently discover task-specific architectures. It encodes MobileNetV2 variants, predicts subnet performance without full evaluation, and alternates search with targeted fine-tuning.

  • NAT overview: NAT initializes an architecture archive, evaluates sampled subnets using inherited supernet weights, and fits an accuracy predictor from architecture–performance pairs.The predictor and user-specified objectives guide subsequent many-objective evolutionary search.
  • Supernet adaptation: NAT repeatedly adds promising architectures to the archive and fine-tunes the corresponding supernet weights, returning both the archive and a task-specific supernet.The algorithm alternates search and adaptation until a preset iteration budget is exhausted.
  • Problem formulation: The NAS problem is formulated as bilevel optimization over architecture a and associated weights w(a), with objectives including validation performance, parameters, MAdds, latency, power, or memory.The inner problem minimizes cross-entropy training loss, while the outer problem optimizes potentially competing objectives.
  • Search space and encoding: The search space uses 22-integer strings: resolution and width multiplier occupy the first two positions, while 20 layer positions encode expansion ratios and kernel sizes.Expansion ratios are, kernel sizes are, and layer settings are sorted by ascending #MAdds.
  • Accuracy predictor: The surrogate accuracy predictor regresses subnet performance directly from encoded architectures, reducing evaluation time from hours or minutes to seconds.The predictor is designed for reliable rank ordering, cross-dataset consistency, and sample efficiency; an RBF ensemble can be learned in under a minute.
  • Many-objective evolutionary search: The evolutionary search improves populations through crossover and mutation, then selects architectures offering the best trade-offs across many objectives.Crossover uniformly inherits integers from parents, while mutation centricity is controlled by η_m: higher values keep offspring nearer their parents.

4 EXPERIMENTAL EVALUATION

NAT is evaluated across eleven diverse image-classification datasets and multiple accuracy–efficiency trade-offs. Across these settings, NATNets generally outperform existing designs, including under mobile constraints, while transferring architecture and weights to new tasks.

  • Evaluation setup: Eleven datasets span large-, medium-, and small-scale multi-class, fine-grained, and texture-classification tasks.ImageNet supplies the supernet training data; the other ten datasets are used for architecture transfer.
  • Evaluation setup: About 6 days of one-time ImageNet supernet preparation amortizes across subsequent dataset and objective transfers.The supernet uses progressive shrinking and is trained on eight V100 GPUs.
  • ImageNet classification: 80.5% top-1 accuracy is achieved by NAT-M4 under the mobile constraint of ≤600M MAdds on ImageNet.NATNets dominate existing manual and NAS designs in both accuracy and MAdds under this setting.
  • ImageNet classification: 2.3% higher top-1 accuracy than MobileNetV3 is achieved by NAT-M1 at equivalent efficiency.NAT-M1 is also 1.5% more accurate than FBNetV2-F4 while matching efficiency.
  • Transfer across datasets: NATNets achieve state-of-the-art performance under mobile settings across the evaluated tasks and are an order of magnitude more efficient in MAdds than existing models.Small-scale fine-grained datasets benefit especially strongly, while fine-grained Cars and Aircraft also improve over conventional fine-tuning.
  • Transfer across datasets: Different datasets produce dissimilar architectures at 350M MAdds, indicating dataset-specific architecture requirements for accuracy–efficiency trade-offs.NAT is reported to generate these task-specific architectures.
  • Multiple objectives: NATNets outperform MobileNetV3 across top-1 accuracy, model size, MAdds, CPU latency, and GPU latency in the three-objective analysis.Compared with MUXNets, NATNets perform favorably on the remaining efficiency metrics except model parameters.
  • Dense prediction: NAT backbones on Cityscapes match Auto-DeepLab mIoU while using 4× fewer MAdds after 4K iterations of fine-tuning.Before fine-tuning, NAT backbones consistently outperform the original ResNet-based BiSeNet backbones.

5 ABLATION STUDY

The ablations examine surrogate predictors, search operators, computational efficiency, and adaptation strategies. They support the use of an RBF ensemble, crossover, and supernet adaptation, especially when target datasets have limited data.

  • Accuracy prediction: The RBF ensemble has higher Spearman rank correlation than alternative predictors across training-sample regimes and datasets.It improves over standalone RBF while balancing prediction quality against the number of training examples.
  • Search efficiency: NAT’s search phase is compared with random search and NSGANet in a controlled bi-objective experiment.The comparison uses mean hypervolume over five runs, where larger hypervolume indicates a better Pareto front.
  • Crossover: Crossover significantly improves evolutionary-search efficiency across ImageNet, CIFAR-10, Oxford Flowers102, and Stanford Cars.The ablation reports median performance over eleven runs and finds deterioration as crossover probability decreases from 0.9 to 0.2.
  • Adaptation strategy: Supernet adaptation yields significantly better final Pareto fronts than adapting individual subnets on four datasets.Additional fine-tuning can match it on larger datasets, but a large gap remains on datasets with few samples per class.
  • Architectural advancement: NAT-M1 is 0.5–1.0% more accurate on ImageNet than compared models using similar or fewer MAdds.This result is reported as part of the architectural-advancement analysis.
  • Architectural advancement: NAT-M1 is evaluated as a drop-in backbone replacement across object detection, semantic segmentation, and instance segmentation.The replacements target EfficientDet-D0, BiSeNet, and YOLACT, respectively.

6 CONCLUSION

The paper introduces NAT for designing task-specific neural networks that trade off multiple objectives. Across eleven diverse image-classification datasets, NAT-produced networks outperform conventional fine-tuning while being orders of magnitude more efficient under mobile settings.

  • NAT designs custom neural network architectures that trade off multiple objectives for a given image classification task.
  • NAT combines a supernet with evolutionary search to design task-specific networks trading off accuracy and computational complexity.
  • Eleven benchmark datasets showed NAT networks outperform conventional fine-tuning-based transfer learning while being orders of magnitude more efficient under mobile settings (≤600M Multiply-Adds).
  • NAT was especially effective on small-scale fine-grained tasks where fine-tuning ImageNet models is ineffective.
  • NAT optimized up to twelve objectives and used subsequent trade-off analysis to identify a single preferred solution.

APPENDIX A RELATION TO EXISTING ONE-SHOT NAS

Existing one-shot NAS trains a supernet before searching, requiring broad simultaneous subnet coverage. NAT instead iteratively uses many-objective search to guide supernet adaptation toward promising task-specific subnets.

  • Existing one-shot NAS separates supernet training and architecture search into sequential stages, using inherited subnet weights during search.
  • OnceForAll still trains its supernet offline from randomly sampled subnets, despite progressive shrinking and full-scale supervision.
  • 634K sampled subnets covered less than 10^-12% of the total subnet volume, while adding search options could require 100x more training epochs.
  • NAT estimates the distribution of optimal subnets from many-objective search results and progressively refines supernet training toward that distribution.
  • The online surrogate approach adaptively learns the objective landscape instead of exhausting the evaluation budget on uniformly sampled solutions initially.

APPENDIX B MANY-OBJECTIVE SELECTION CONTINUED

The appendix motivates reference-point selection for many-objective optimization and contrasts online and offline surrogate modeling. Reference-point selection achieves consistently lower IGD across objective counts.

  • Domination ranks solutions, but solutions in the same non-dominated rank can become difficult to distinguish as objectives increase.
  • The online surrogate initially learns a coarse landscape with partial budget and refines it during optimization, unlike offline uniform sampling.
  • Reference points provide predefined targets that guide selection when domination cannot distinguish non-dominated solutions.
  • Hypervolume is computationally infeasible to calculate under large numbers of objectives.
  • Reference-point selection produced consistently lower IGD values across different numbers of objectives than domination-based selection.

APPENDIX C CHOOSING BEST TRADE-OFF SOLUTION

NAT evaluates candidate solutions by their trade-offs among multiple objectives and can select a preferred solution when users provide no preferences. The trade-off measure focuses on losses required for gains among neighboring solutions.

  • The many-objective evolutionary algorithm produces N solutions in which gains in one objective require losses in at least one other objective.
  • Trade-off is calculated as average loss per unit average gain among m nearest neighbors based on normalized Euclidean distance.
  • A solution with the highest trade-off value causes the largest average loss in some objectives for a unit average gain in others.
  • When the highest trade-off value is statistically much larger than others, that solution is preferred if users provide no preferences.

APPENDIX D COMPARISON TO EXISTING CONVNETS

NATNets improve the accuracy–computation trade-off over existing convolutional networks, including state-of-the-art mobile performance on ImageNet.

  • 20x: NAT-M1 is more accurate and more efficient in MAdds than ResNet-50.
  • 21x: NAT-M4 is more accurate and more efficient in MAdds than Inception-ResNet-v2.
  • 80.5% top-1 accuracy: NAT-M4 achieves a new state-of-the-art on ImageNet under the 600M MAdds mobile setting.

APPENDIX E ARCHITECTURE VISUALIZATION

NAT’s multi-objective search produces non-dominated architectures whose design choices vary across datasets, even under identical optimization objectives.

  • Non-dominated solutions expose trade-offs between objectives and can reveal design principles for constructing dataset-specific architectures.
  • 192: Oxford-IIIT Pets and STL-10 most frequently select the lowest searched input resolution, whereas FGVC Aircraft and Food-101 most frequently select 256.
  • Architecture choices such as input resolution, width multiplier, and layer settings differ across datasets under the same objectives.

APPENDIX F SCALABILITY TO OBJECTIVES CONTINUED

NAT scales to a 12-objective search across 11 datasets and identifies a preferred compromise architecture that performs strongly across datasets against several baselines.

  • 12-objective optimization combines top-1 accuracy on 11 datasets with #MAdds, and NAT supports this large-scale search.
  • The 12-objective search reveals information that a low-dimensional optimization may not expose.
  • During this experiment, NAT evaluates inherited weights from dataset-specific supernets and excludes further supernet adaptation.
  • 45: NAT obtains 45 non-dominated architectures, from which trade-off analysis selects a solution balancing all 12 objectives.
  • The selected NATNet achieves better accuracy on all datasets with similar or fewer #MAdds than EfficientNet-B0, MobileNetV2, NASNet-A, and ResNet-50.
Loading 2005.05859v2…