PaperlayerView exact revision on arXiv

Source-linked AI summary

How to train your MAML

Antreas Antoniou, Harrison Edwards, Amos Storkey

arXiv:1810.09502v3cs.LGstat.ML

TL;DR

Few-shot learning requires effective adaptation from very limited data, but MAML can be unstable, hyperparameter-sensitive, and computationally expensive. The paper proposes MAML++, which combines multi-step loss optimization with per-step batch-normalization parameters and learned inner-loop rates. MAML++ improves stability, efficiency, convergence, and generalization, achieving state-of-the-art results across Omniglot and Mini-Imagenet few-shot tasks.

  • Problem

    MAML is powerful for few-shot learning but can be unstable, sensitive to architecture and hyperparameters, and computationally expensive during training and inference.

  • Method

    MAML++ modifies MAML with multi-step target-loss optimization, per-step batch-normalization statistics and weights, and learned inner-loop learning rates.

  • Results

    MAML++ sets a new state of the art across established few-shot tasks on Omniglot and Mini-Imagenet, including 52.15% and 68.32% on Mini-Imagenet 5-way 1-shot and 5-shot tasks.

  • Takeaways & Limitations

    Learning per-step rates, batch-normalization parameters, and per-step target losses appears key to fast, automatic, stable, and strongly generalizable few-shot learning.

  • Takeaways & Limitations

    MAML training can remain unstable because gradients are backpropagated through repeated network updates, especially in architectures without skip-connections, while batch normalization without accumulated statistics reduces effectiveness.

Abstract

from arXiv · show

The field of few-shot learning has recently seen substantial advancements. Most of these advancements came from casting few-shot learning as a meta-learning problem. Model Agnostic Meta Learning or MAML is currently one of the best approaches for few-shot learning via meta-learning. MAML is simple, elegant and very powerful, however, it has a variety of issues, such as being very sensitive to neural network architectures, often leading to instability during training, requiring arduous hyperparameter searches to stabilize training and achieve high generalization and being very computationally expensive at both training and inference times. In this paper, we propose various modifications to MAML that not only stabilize the system, but also substantially improve the generalization performance, convergence speed and computational overhead of MAML, which we call MAML++.

1 INTRODUCTION

Few-shot learning targets learning new concepts from only a handful of examples, motivating meta-learning approaches that transfer knowledge across tasks. MAML learns an initialization for rapid adaptation, while MAML++ extends it with stability, efficiency, and generalization improvements.

  • Few-shot learning aims to learn new concepts from usually 1–5 samples per concept, unlike standard deep networks that generally require thousands of examples.
  • Meta-learning automatically transfers knowledge across similar tasks, avoiding the time-consuming and inefficient manual fine-tuning of each new task.
  • MAML learns base-model initialization parameters that can achieve strong validation performance after a small number of gradient updates on a task’s training set.
  • MAML++ improves MAML with robust training, automatic inner-loop hyperparameter learning, greater computational efficiency, and significantly improved generalization across Omniglot and Mini-Imagenet.

2 RELATED WORK

Set-to-set few-shot learning frames adaptation as learning from support sets and evaluation on target sets across meta-training, validation, and test tasks. Subsequent methods include matching-based, recurrent-gradient, MAML, and Meta-SGD approaches with different adaptation mechanisms and computational trade-offs.

  • Set-to-set few-shot learning uses support sets for task-level learning and target sets to evaluate the adapted base model across meta-training, validation, and test tasks.
  • Matching Networks learn to match target items to support items using cosine distance and differentiable embedding functions.
  • The gradient-conditional meta-learner LSTM learns how to update a base learner and applies a support-set gradient update before target-set prediction.
  • MAML replaces the meta-learner LSTM with batch stochastic gradient descent and increases base-model update steps, improving generalization in Omniglot and Mini-Imagenet experiments.
  • Meta-SGD learns parameter-specific learning rates and update directions alongside initialization parameters, improving generalization with one inner-loop step but increasing parameters and computational overhead.

3 MODEL AGNOSTIC META LEARNING

MAML learns an initialization that adapts to few-shot tasks through inner-loop updates and outer-loop optimization. Its simplicity and performance are offset by instability, expensive second-order computation, and several batch-normalization and learning-rate limitations.

  • MAML framework: MAML learns initial network parameters that perform well on a task after a small number of gradient updates on its support set.The adapted model is evaluated on the task’s target set.
  • MAML framework: The inner loop updates task-specific parameters using support-set loss, while the outer loop optimizes the initialization using target-set losses across tasks.The initialization contains across-task knowledge.
  • MAML problems: MAML can become unstable because outer-loop backpropagation through an unfolded inner loop compounds gradient explosions and diminishing-gradient problems.The issue is aggravated by a deep unfolded network without skip-connections.
  • MAML problems: MAML’s outer-loop optimization requires expensive second-order gradients, while first-order approximations can negatively affect final generalization error.The paper identifies the need to reduce computation without sacrificing generalization performance.
  • MAML problems: MAML’s batch normalization uses several restrictive choices, including non-accumulated statistics and shared biases across inner-loop steps.The paper argues that accumulated statistics can improve convergence speed, stability, and generalization, while shared biases can restrict performance.
  • MAML problems: A shared fixed inner-loop learning rate requires costly dataset-specific hyperparameter searches and limits parameter-specific update rates.Meta-SGD addresses manual learning-rate search but increases computational effort and memory usage by learning rates per parameter.

4 STABLE, AUTOMATED AND IMPROVED MAML

MAML++ addresses instability, computational cost, and limited automation in MAML through a set of targeted training, normalization, learning-rate, and derivative-order modifications. Across Omniglot and Mini-Imagenet, these changes improve convergence and test performance, while multi-step loss improves training stability.

  • Proposed improvements: MAML++ combines six proposed methodologies to address MAML’s optimization, normalization, learning-rate, and computational issues.The methods include cosine annealing, multi-step loss optimization, derivative-order annealing, per-step batch-normalization statistics, per-step weights and biases, and learned per-layer per-step learning rates.
  • Optimization stability: Multi-step loss optimization minimizes a weighted sum of target-set losses after every support-set update, improving gradient propagation through the inner loop.Weights v_i determine the importance of each target-set loss, and intermediate steps receive direct and indirect gradient signals.
  • Computational efficiency: Derivative-order annealing speeds the first 50 epochs while retaining second-order training for the strong generalization performance associated with second-order gradients.The experiments also report no exploding or diminishing gradients in derivative-order annealing experiments, unlike second-order-only experiments.
  • Normalization: Per-step batch-normalization statistics and weights and biases avoid sharing normalization parameters across increasingly different inner-loop models.The proposed BNRS and BNWB methods use step-specific statistics and parameters to accommodate changing feature distributions during fast adaptation.
  • Omniglot results: 99.47% is achieved on 5-way 1-shot Omniglot, while MAML++ reaches 97.76% and 99.33% on 20-way 1-shot and 5-shot tasks.The authors also report improved convergence speed and substantially improved training stability from multi-step loss optimization, while noting difficulty reproducing the original 20-way 1-shot MAML result.
  • Mini-Imagenet results: 52.15% and 68.32% are achieved on Mini-Imagenet 5-way 1-shot and 5-shot tasks, respectively, with 51.05% obtained using only one inner-loop step.The paper reports faster convergence than MAML and lower cost from derivative-order annealing and fewer inner-loop steps.

5 CONCLUSION

MAML++ addresses MAML’s sensitivity, generalization, stability, and speed through multiple modifications, achieving state-of-the-art few-shot results across Omniglot and Mini-Imagenet.

  • MAML++ reduces inner-loop hyperparameter sensitivity, improves generalization error, and stabilizes and accelerates MAML.The approach is designed to make few-shot learning more automatic and computationally efficient.
  • MAML++ sets a new state of the art across all few-shot tasks on Omniglot and Mini-Imagenet.
  • Learning per-step learning rates, batch normalization parameters, and per-step target losses appears key to MAML++’s speed, automation, and generalization.

A ADDITIONAL RESULTS

Additional results show that MAML++ reduces training-iteration time relative to MAML and matches or improves MAML’s Omniglot performance while using fewer parameters than Meta-SGD.

  • MAML++ requires less time per training iteration than MAML across variants with 1 to 5 inner-loop steps.The comparison covers model variants spanning 1 to 5 inner-loop steps, despite MAML++ using more parameters and computations.
  • More inner-loop steps require more computation but provide better generalization performance.
  • MAML++ matches or improves on MAML across all reported Omniglot 5-way cases.
  • MAML++ performs close to Meta-SGD while using fewer additional parameters and therefore smaller training and testing times.Meta-SGD uses about 40K extra parameters, whereas MAML++ uses one extra parameter per layer per step for LSLR and additional BNWB parameters.
Loading 1810.09502v3…