PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-Learning of Neural Architectures for Few-Shot Learning

Thomas Elsken, Benedikt Staffler, Jan Hendrik Metzen, Frank Hutter

arXiv:1911.11090v3cs.LGstat.ML

TL;DR

Few-shot learning makes conventional NAS difficult because many related tasks provide little data and compute, while prior methods commonly use fixed architectures. MetaNAS jointly meta-learns architecture and weights, then adapts architectures to novel tasks with few data and gradient steps; its DARTS–REPTILE combination achieves strong benchmark results, including 63.1% and 79.5% MiniImagenet accuracy in 1-shot and 5-shot settings.

  • Problem

    NAS depends on substantial labeled data and computation, making architecture search difficult across many few-shot tasks with limited per-task resources.

  • Method

    MetaNAS jointly meta-learns a neural meta-architecture and weights, adapting task-specific architectures with few datapoints and a few task-optimizer steps.

  • Results

    63.1% MiniImagenet accuracy in the 1-shot, 5-way setting and 79.5% in the 5-shot, 5-way setting are reported after scaling the found architectures.

  • Takeaways & Limitations

    MetaNAS provides task-dependent architecture adaptation without retraining after pruning and is compatible with model-agnostic meta-learning and gradient-based NAS methods.

  • Takeaways & Limitations

    The framework is evaluated beyond few-shot classification only as future work, and the authors suggest more sophisticated meta-learning and NAS methods could improve results.

Abstract

from arXiv · show

The recent progress in neural architecture search (NAS) has allowed scaling the automated design of neural architectures to real-world domains, such as object detection and semantic segmentation. However, one prerequisite for the application of NAS are large amounts of labeled data and compute resources. This renders its application challenging in few-shot learning scenarios, where many related tasks need to be learned, each with limited amounts of data and compute time. Thus, few-shot learning is typically done with a fixed neural architecture. To improve upon this, we propose MetaNAS, the first method which fully integrates NAS with gradient-based meta-learning. MetaNAS optimizes a meta-architecture along with the meta-weights during meta-training. During meta-testing, architectures can be adapted to a novel task with a few steps of the task optimizer, that is: task adaptation becomes computationally cheap and requires only little data per task. Moreover, MetaNAS is agnostic in that it can be used with arbitrary model-agnostic meta-learning algorithms and arbitrary gradient-based NAS methods. %We present encouraging results for MetaNAS with a combination of DARTS and REPTILE on few-shot classification benchmarks. Empirical results on standard few-shot classification benchmarks show that MetaNAS with a combination of DARTS and REPTILE yields state-of-the-art results.

1. Introduction

NAS brings automated architecture design to vision tasks but depends on substantial labeled data and computation, limiting its use across few-shot task distributions. METANAS jointly meta-learns architectures and weights, enabling task-specific architecture adaptation from few data and few optimizer steps.

  • Motivation: NAS has advanced across vision tasks but requires large labeled datasets and considerable compute resources.These requirements make NAS difficult to apply when learning many related tasks with limited data and compute.
  • Motivation: Few-shot learning seeks rapid adaptation to novel tasks from few examples, and prior model-agnostic methods typically meta-learn weights for fixed architectures.This setting uses experience from related tasks to improve learning of new tasks.
  • METANAS: METANAS jointly meta-learns a neural architecture and its weights, rather than restricting meta-learning to fixed architectures.It combines model-agnostic gradient-based meta-learning with gradient-based NAS.
  • METANAS: METANAS adapts a meta-architecture to task-dependent architectures using few labeled datapoints and only a few task-optimizer steps.The resulting architectures are adapted separately to novel tasks while originating from a jointly meta-learned meta-architecture.
  • METANAS: Soft-pruning with temperature annealing avoids retraining task-dependent architectures after pruning and reduces pruning-related accuracy drops.The mechanism lets architecture parameters converge toward hard-pruned architectures while weights adapt to the pruning.
  • Results: METANAS is compatible with arbitrary gradient-based model-agnostic meta-learning and continuous-relaxation NAS methods, including REPTILE and DARTS.The REPTILE–DARTS combination yields state-of-the-art results on Omniglot and MiniImagenet.

2. Related Work

Related work spans few-shot meta-learning, automated architecture search, and differentiable NAS. METANAS builds on gradient-based weight adaptation and continuous architecture relaxation while addressing the computational and retraining costs of prior NAS approaches.

  • Few-Shot Learning via Meta-Learning: Few-shot learning trains models to solve new tasks from a few examples, often using meta-learning across varied tasks to accelerate adaptation.Model-agnostic methods meta-learn initial weights that adapt through a few gradient-descent steps.
  • Neural Architecture Search: NAS automatically designs neural networks but traditional reinforcement-learning and evolutionary approaches can require training hundreds or thousands of architectures from scratch.Weight sharing, network morphisms, and multifidelity optimization improve efficiency but may remain limited to relatively small problems.
  • Differentiable NAS: DARTS continuously relaxes the architecture space by representing each layer as a weighted sum of candidate operations optimized with gradient descent.The architecture parameters are real-valued and define a mixed operation containing the candidate operations.
  • Differentiable NAS: DARTS alternates optimization of model weights and architecture parameters, then hard-prunes all but the most important incoming operations for each intermediate node.This converts the one-shot model into a discrete architecture after the search phase.
  • Differentiable NAS: Hard-pruning can sharply reduce accuracy and require retraining the pruned weights.One reported example drops from 88% one-shot accuracy to 56% after pruning.
  • NAS for Meta-Learning: Prior NAS for few-shot learning either repeatedly reruns meta-training at high computational cost or searches for one architecture shared across tasks.The cited progressive-search approach requires more than a hundred GPU days and does not learn task-dependent architectures.

3. Marrying Gradient-based Meta-Learning and Gradient-based NAS

METANAS integrates gradient-based neural architecture search with model-agnostic meta-learning, jointly learning architecture parameters and weights so both can adapt to new few-shot tasks. Its framework searches architectures in one meta-learning run and is compatible with arbitrary gradient-based NAS and meta-learning methods.

  • Joint Meta-Learning: METANAS meta-learns architecture parameters αmeta together with corresponding weights wmeta across a distribution of few-shot tasks.At meta-testing, both are adapted to a new task from a few labeled examples.
  • Problem Setup: Few-shot learning uses n-way, k-shot tasks sampled from training and test task distributions, unlike single-task supervised learning with large training data.Typical settings use n ∈ {5, 20} classes and k ∈ {1, 5} examples per class.
  • Task Optimization: The task learner performs k gradient-based NAS steps inspired by DARTS, updating weights and architecture parameters with task-specific learning rates.The architecture is represented by real-valued parameters α, making the meta-objective differentiable.
  • Meta-Optimization: REPTILE is used in the experiments because of its conceptual simplicity and computational efficiency compared with MAML.The framework can use different gradient-based meta-learning and NAS algorithms, although the experiments use the same meta-learning algorithm for both parameter sets.
  • Efficiency and Flexibility: Integrating NAS directly into meta-learning searches architectures in a single meta-learning run instead of fully meta-learning hundreds of proposed architectures.The framework is agnostic to the choice of gradient-based NAS and meta-learning methods.

4. Task-dependent Architecture Adaptation

METANAS adapts a meta-learned architecture to each novel task during meta-testing, then uses soft pruning to make the resulting architecture sparse without retraining. The modifications address DARTS’s non-sparse task-dependent architectures and preserve performance after pruning.

  • Architecture Adaptation: NAS as the task optimizer enables task-dependent architecture adaptation during meta-testing, unlike methods that keep the architecture fixed or share one architecture across tasks.The adapted architecture starts from the jointly meta-learned architecture and weights.
  • Motivation for Soft Pruning: Vanilla DARTS produces non-sparse task-dependent architectures, so hard pruning can substantially deteriorate performance and require retraining.This is especially costly in few-shot learning because each task-dependent architecture would otherwise need meta-retraining.
  • Soft-Pruning of Operations: METANAS anneals operation-mixture temperatures toward zero, allowing mixture weights to approach one-hot selections while operation weights adapt during soft pruning.This sparsifies the operations within each mixed operation.
  • Soft-Pruning of Input Nodes: Because operation sparsity alone leaves every prior node connected, METANAS additionally sparsifies input-node combinations rather than selecting only one input.The method weights combinations of k inputs, supporting a predefined number of inputs such as the default k = 2.
  • Pruning Outcome: The two modifications produce task-dependent architectures that can be hard-pruned without notable performance loss or retraining.In practice, a threshold such as α̂ ≥ 0.01 retains weights, and allowing a few weights above zero can outperform strict one-hot selection.

5. Experiments

Experiments evaluate METANAS on Omniglot and MiniImagenet few-shot classification, comparing task-adapted and scaled fixed architectures with REPTILE and other meta-learning methods. METANAS matches or exceeds relevant baselines while requiring substantially less computation than AutoMeta.

  • Evaluation: METANAS is evaluated on Omniglot and MiniImagenet in n-way, k-shot few-shot image-recognition tasks.Tasks are generated by sampling n classes and k examples per class.
  • Comparison under the same meta-learning algorithm: METANAS always outperforms the original REPTILE architecture and is on-par with AutoMeta across the reported Omniglot and MiniImagenet settings.On Omniglot, METANAS is slightly inferior to AutoMeta in 1-shot, 20-way and on-par in 5-shot, 20-way; on MiniImagenet, it outperforms AutoMeta.
  • Efficiency: More than 10x lower computational cost distinguishes METANAS from AutoMeta: approximately one week on one GPU versus about 100 GPU days.Both comparisons use a single meta-training run for METANAS, while the AutoMeta authors report costs in the order of 100 GPU days.
  • Task-dependent architectures: METANAS adapts architectures to tasks, typically producing only a few different architectures for MiniImagenet and Omniglot.Figure 4 reports the most common normal and reduction cells used for evaluation, while Figure 5 illustrates model-size variation across experiments and seeds.
  • Scaling up architectures: The scaled METANAS architecture has approximately 1.1 million parameters and is retrained with more meta-epochs and stronger regularization.This fixed-architecture comparison uses the most common task-dependent architecture extracted after METANAS runs.
  • Comparison to other meta-learning algorithms: METANAS significantly outperforms other initial-parameter meta-learning methods on MiniImagenet and is on-par with MAML++ on Omniglot.Against other meta-learning approaches, it is on-par with or better than them while using about 1 million parameters instead of more than 10 million for several alternatives.

6. Conclusion

The paper presents METANAS as a full integration of gradient-based meta-learning and NAS, enabling few-shot adaptation of both architecture and weights. On standard benchmarks, scaled METANAS reaches state-of-the-art MiniImagenet performance, while the framework remains open to stronger component methods and broader multi-task applications.

  • Conclusion: METANAS fully integrates gradient-based meta-learning with neural architecture search, jointly meta-learning architecture and weight parameters.The method adapts to task-dependent architectures using few labeled datapoints and only a few gradient-descent steps.
  • Conclusion: METANAS is on-par or better than other NAS-based few-shot methods while being significantly more efficient.The conclusion also reports superiority over simple CNNs commonly used in few-shot learning.
  • Conclusion: 63.1% accuracy in the 1-shot, 5-way MiniImagenet setting and 79.5% in the 5-shot, 5-way setting are reported as state-of-the-art results.These results follow scaling of the found architectures.
  • Future directions: The framework is agnostic to the meta-learning and differentiable NAS algorithms, and the authors identify stronger methods and broader multi-task problems as future directions.The authors specifically mention MAML++ and ProxylessNAS as possible replacements.

A.1. Comparison of different sparsification strategies and impact on pruning

The appendix compares three DARTS sparsification strategies for pruning: no sparsification, operation-only sparsification, and the proposed joint sparsification of operations and inputs. The strategies are evaluated by the accuracy drop from one-shot to pruned models on CIFAR-10.

  • Strategies: The comparison includes vanilla DARTS with no sparsification, operation-only sparsification, and sparsification of both operations and inputs.The third strategy is the one proposed in the paper.
  • Evaluation: The evaluation measures the accuracy drop after pruning from the one-shot model to the pruned model on single-task CIFAR-10 NAS.Experiments use DARTS default hyperparameters and report mean ± SEM.
  • Results: Without sparsification, pruning causes a significant accuracy drop, almost reducing performance to chance level.The one-shot model is pruned using the procedure proposed by Liu et al.

A.2. Detailed experimental setup and hyperparameters

The detailed setup uses REPTILE and DARTS implementations with Torchmeta data splits and validation-based hyperparameter selection. Training includes a NAS warm-up phase, while the scaled experiment increases training duration and model size and adds regularization.

  • Implementation: The implementation is based on REPTILE and DARTS code, uses Torchmeta data loaders and splits, and follows the REPTILE evaluation setup.These choices define the shared experimental infrastructure.
  • Hyperparameters: Hyperparameters are selected by random search around REPTILE defaults using a validation split of the training data.The listed Section 5.1 hyperparameters are shared across n-shot and k-way settings, with dataset-specific exceptions indicated by paired values.
  • Training schedule: Section 5.1 trains all models for 30,000 meta epochs, with METANAS delaying architectural adaptation for the first 15,000 epochs.The warm-up phase is used to help avoid unstable behavior in gradient-based NAS.
  • Scaled experiment: Section 5.2 uses 100,000 meta epochs, increases channels per layer from 28 to 96, and uses five cells instead of four.The second and fourth cells remain reduction cells, while the others are normal cells.
  • Scaled experiment: The scaled experiment adds weight decay of 10^-4 and DropPath regularization to the default REPTILE meta-learning hyperparameters.These changes accompany the larger model and longer training schedule.

A.3. Motivation of meta-learning algorithm Ψ

The proposed meta-learning updates extend familiar MAML and REPTILE updates so that architectural parameters receive analogous updates alongside weights. The REPTILE-based update is motivated by improved same-task gradient alignment and generalization.

  • The 50+50 schedule adapts both weights and architecture for the first 50 steps, then adapts only weights for the next 50.
  • Equations (8) and (9) extend MAML and REPTILE updates to architectural parameters.They apply the same updates to architecture variables as to model weights.
  • The MAML extension corresponds to one step of SGD on the meta-objective.
  • REPTILE was shown to maximize the inner product between gradients from different batches of the same task, improving generalization.

A.4. Additional plots

Figure 5 presents histograms of model sizes, measured by parameter count, after task adaptation across different few-shot learning problems.

  • Figure 5 shows model-size histograms after task adaptation for different few-shot learning problems.Model size is measured by the number of parameters.
Loading 1911.11090v3…