Source-linked AI summary
Transfer Learning with Neural AutoML
Catherine Wong, Neil Houlsby, Yifeng Lu, Andrea Gesmundo
TL;DR
Neural AutoML automates architecture design but requires substantial computation and repeated search for each task. Transfer Neural AutoML learns shared and task-specific search priors across tasks, then transfers its controller to new tasks. Across language and image classification, it reduces convergence time over single-task training by over an order of magnitude on many tasks.
Problem
Neural AutoML has high computational cost because it trains many networks and repeats the search for every new task.
Method
Transfer Neural AutoML multitask-trains an RL-based architecture-search controller, learning shared priors and task-specific embeddings before transferring it to new tasks.
Results
Transfer Neural AutoML reduces convergence time by over an order of magnitude on most tasks and saves 10s of CPU hours for each transferred task.
Takeaways & Limitations
The controller provides a reusable search strategy that can accelerate network design across language and image tasks.
Takeaways & Limitations
Meta overfitting can occur because the controller is trained on validation sets, and Transfer AutoML exhibits more overfitting than single-task training in some cases.
Abstract
from arXiv · showhide
We reduce the computational cost of Neural AutoML with transfer learning. AutoML relieves human effort by automating the design of ML algorithms. Neural AutoML has become popular for the design of deep learning architectures, however, this method has a high computation cost. To address this we propose Transfer Neural AutoML that uses knowledge from prior tasks to speed up network design. We extend RL-based architecture search methods to support parallel training on multiple tasks and then transfer the search strategy to new tasks. On language and image classification tasks, Transfer Neural AutoML reduces convergence time over single-task training by over an order of magnitude on many tasks.
1 Introduction
Neural AutoML automates deep-learning architecture design but is computationally expensive, especially when search must be repeated across tasks. Transfer Neural AutoML addresses this cost by reusing knowledge from prior tasks to accelerate network design.
- Neural AutoML uses deep reinforcement learning to optimize architectures and has produced networks competitive with human-designed models on challenging image-classification tasks.
- 800 concurrent GPUs were reported for training Neural AutoML on CIFAR-10, illustrating its substantial computational cost.
- Transfer Neural AutoML learns shared and task-specific priors through multitask search, then transfers the controller to new tasks.
- Transfer Neural AutoML reduces convergence time by over an order of magnitude on most language and image tasks and saves 10s of CPU hours per transferred task.
2 Methods
The method extends RL-based neural architecture search into multitask training by conditioning a shared controller on learned task embeddings. A pretrained controller is then adapted to new tasks using a new task embedding and continued search.
- 2.1 Neural Architecture Search: The controller RNN generates discrete design actions autoregressively, trains each resulting child model, and uses validation performance as a policy-gradient reward.
- 2.2 Multitask Training: Multitask Neural AutoML searches several tasks simultaneously in a shared space, learning broadly applicable architectural and hyperparameter priors.
- 2.2 Multitask Training: Each task receives a trainable embedding that conditions the shared controller, while the remaining controller parameters are shared across tasks.
- 2.2 Multitask Training: Task-specific advantage normalization rescales reward distributions to comparable means and variances, balancing policy-gradient updates across tasks.
- 2.2 Multitask Training: Transfer reloads the pretrained controller, adds a randomly initialized embedding for the new task, and jointly updates the controller and new embedding during resumed search.
3 Related Work
The work builds on architecture search, optimization, transfer learning, and multitask-learning research. Its closest distinction from prior NAS transfer is transferring the controller itself rather than only a child architecture.
- Related AutoML approaches include random search, parameter modeling, meta-learned initialization, tree search, learned optimizers, and neuro-evolution.
- Prior NAS work transferred a child architecture from CIFAR-10 to ImageNet, whereas this work transfers the controller model itself.
- Other methods reduce search cost through aggressive exploration, incremental architecture transformation, progressive complexity, or child-model weight sharing.
- The approach also relates to transfer learning, meta-learning, and simultaneous multitask training for learning representations across tasks.
4 Experiments
Experiments compare transfer, single-task, and random-search controllers across language and image tasks, showing faster search with competitive final models but important overfitting and task-distance limits.
- Experimental setup: Transfer Neural AutoML uses two-tower feedforward child networks and searches shared architecture, embedding, optimization, regularization, and training-step choices.The controller evaluates three strategies: transfer from multitask pretraining, single-task training from scratch, and uniform random search.
- Results: T-AutoML reaches the target validation reward fastest on 12 of 13 datasets, with an order-of-magnitude speed-up in 9 cases.The target matches each dataset’s random-search validation accuracy-top10 under the stated trial budget.
- Results: At 500 trials, T-AutoML has the best test accuracy-top10 on all but one dataset and beats single-task AutoML on 10 of 13 datasets.It ties single-task on one dataset and loses on two; transfer often produces stronger models earlier even when its final model is not best.
- Analysis: Multitask training alone yields limited final-accuracy gains over single-task training, while learned task embeddings group tasks with shared architecture and hyperparameter choices.The controller assigns distinct two-layer/64-unit/ReLU and one-layer/256-unit/Swish configurations to different task clusters.
- Computational cost: The median computational saving is 30 CPU-hours per task at 500 trials, while the multitask controller requires 15 hours on 100 CPUs and needs over 1.05M new tasks to amortize that cost at median speedup.The median single-trial cost is 268 seconds, and the amortization estimate assumes the pretraining tasks’ models are not otherwise needed.
- Analysis: Validation overfitting can widen the validation–test gap, especially on tiny validation sets, and transfer may overfit more because it converges earlier.On Prog Opinion, validation performance keeps rising while test performance does not; the validation set contains only 116 examples.
- Task distance: Transfer performance is strongest across similar tasks but adapts to distant languages after sufficient training, reaching 79.8% versus 79.4% for random search and 78.1% for single-task training.The controller initially favors English embeddings, then switches to Spanish embeddings around trial 2000.
5 Conclusion
Transfer Neural AutoML addresses Neural AutoML’s high computational cost by transferring controller knowledge across tasks, while identifying robustness and generalization as remaining challenges.
- Transfer learning of the controller yields large reductions in convergence time across many datasets.The paper highlights improved efficiency while noting future work on broader search spaces, cross-modal transfer, evaluation noise, and meta-overfitting.
Supplementary Material for Transfer Learning with Neural AutoML
The supplementary material documents the search space, pretrained input modules, task datasets, and learning-curve evaluations for Transfer and Multitask Neural AutoML.
- The search space spans input embeddings, fine-tuning, network depth and width, activations, normalization, dropout, learning rates, regularization, and training-sample counts.
- Text inputs use pretrained embedding tables with tokenization, normalization, and mean bag-of-words aggregation.
- Image inputs can use state-of-the-art architectures pretrained on ImageNet, truncated before the final logits layer.
- The NLP task statistics and references are provided in Table 4, while image-classification task statistics and references are provided in Table 5.
- Figures 1–4 report validation and test learning curves for transfer learning and multitask training as trial counts increase.