Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Chelsea Finn; Pieter Abbeel; Sergey Levine

Source-linked AI summary

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Chelsea Finn, Pieter Abbeel, Sergey Levine

arXiv:1703.03400v3Mar 9, 2017cs.LGcs.AIcs.CVcs.NE

TL;DR

Few-shot learners must adapt to new, varied tasks from little data without overfitting. MAML learns gradient-friendly initial parameters, achieving state-of-the-art classification while also supporting regression and faster reinforcement-learning fine-tuning.

Problem
Meta-learning seeks general mechanisms that adapt quickly to varied new tasks from few examples while integrating prior experience without overfitting.
Method
MAML trains model initial parameters so a small number of gradient updates on limited new-task data yields rapid adaptation.
Results
MAML achieves state-of-the-art few-shot image classification, produces good few-shot regression results, and accelerates policy-gradient reinforcement-learning fine-tuning.
Takeaways & Limitations
A learned initialization supports adaptable models across classification, regression, and reinforcement learning using gradient-based training.
Takeaways & Limitations
MAML incurs significant computational expense because its meta-gradient requires backpropagating through second derivatives.

Abstract

from arXiv · show

We propose an algorithm for meta-learning that is model-agnostic, in the sense that it is compatible with any model trained with gradient descent and applicable to a variety of different learning problems, including classification, regression, and reinforcement learning. The goal of meta-learning is to train a model on a variety of learning tasks, such that it can solve new learning tasks using only a small number of training samples. In our approach, the parameters of the model are explicitly trained such that a small number of gradient steps with a small amount of training data from a new task will produce good generalization performance on that task. In effect, our method trains the model to be easy to fine-tune. We demonstrate that this approach leads to state-of-the-art performance on two few-shot image classification benchmarks, produces good results on few-shot regression, and accelerates fine-tuning for policy gradient reinforcement learning with neural network policies.

1. Introduction

The paper introduces a model- and task-agnostic meta-learning algorithm that trains models to adapt quickly to new tasks with few gradient updates and limited data. It applies across architectures and learning settings, including classification, regression, and policy-gradient reinforcement learning.

Motivation: The method targets rapid adaptation from only a few examples while allowing continued adaptation as more data becomes available.This goal is motivated by fast human learning from limited examples and brief experience.
Contribution: The algorithm applies directly to any model trained with gradient descent, with minimal modification across architectures and problem settings.The authors focus on deep neural networks while illustrating flexibility across different architectures and learning problems.
Method: It trains model parameters so a small number of gradient updates, potentially a single step, produces good results on a new task.The process can be interpreted as building an internal representation broadly suitable for many tasks.
Evaluation: The authors demonstrate the approach with fully connected and convolutional networks in few-shot regression, image classification, and reinforcement learning.These demonstrations cover multiple model types and distinct application domains.

2. Model-Agnostic Meta-Learning

MAML trains model parameters across tasks so that one or a small number of gradient updates using few examples enables rapid adaptation and effective performance on new tasks. The method applies to parametrized models trained with gradient-based techniques and optimizes the pre-adaptation parameters through post-update performance.

Problem setup: Few-shot meta-learning trains a model across tasks so it can quickly adapt to new tasks using only a few datapoints and training iterations.The framework treats entire tasks as training examples during meta-learning.
Problem setup: MAML samples tasks from p(T), adapts parameters using K examples and task feedback, then improves the model based on performance on new samples from each task.This training procedure repeatedly adapts and tests across sampled tasks.
Model agnosticism: The method makes no assumption about model form beyond parametrization by θ and a sufficiently smooth loss for gradient-based learning.The meta-gradient requires an additional backward pass through f to compute Hessian-vector products; a first-order approximation is also considered.
Adaptation rule: For a task Ti, one adaptation step computes θ′_i = θ − α∇θL_Ti(fθ), while α may be fixed or meta-learned; multiple updates are a straightforward extension.The model is represented by a parametrized function fθ, and adaptation uses gradient descent updates.
Meta-objective: MAML optimizes the pre-adaptation parameters θ across tasks, although the objective is evaluated using updated parameters θ′, so few gradient steps produce effective task behavior.Meta-optimization is performed with stochastic gradient descent using meta step size β.

3. Species of MAML

MAML applies the same gradient-based adaptation mechanism to supervised learning and reinforcement learning, despite differences in losses and data generation. Supervised tasks use few input/output examples, while RL tasks adapt policies from limited trajectories using policy-gradient estimates.

Overview: MAML instantiations cover supervised and reinforcement learning, applying the same adaptation mechanism despite differences in loss functions and data generation.The section presents supervised and reinforcement-learning variants within a common meta-learning framework.
Supervised Learning: Supervised few-shot tasks learn new functions from limited input/output pairs, including classification and regression problems.Classification uses cross-entropy, while regression commonly uses mean-squared error.
Supervised Learning: K-shot classification provides K input/output pairs per class, totaling NK data points for N-way classification.These task losses can be inserted directly into the meta-learning equations.
Reinforcement Learning: RL few-shot learning adapts a policy to a new task from limited experience, with tasks represented as Markov decision processes whose dynamics may vary.The policy maps states to action distributions, and the task loss corresponds to negative reward.
Reinforcement Learning: Policy-gradient methods estimate both adaptation and meta-optimization gradients because expected rewards are generally nondifferentiable under unknown dynamics.Each additional adaptation step requires new samples because policy gradients are on-policy.

4. Related Work

The paper situates its method within meta-learning and few-shot learning, contrasting gradient-based fast adaptation with learned update rules, metric-based comparison, recurrent memory, and pretrained initialization approaches.

Meta-learning: Meta-learning includes few-shot learning, with one popular approach training a meta-learner to update the learner model’s parameters.This line of work includes methods for learning to optimize deep networks.
Few-shot learning: Few-shot methods address tasks including generative modeling and image recognition, with classification approaches learning comparisons in a metric space.Examples include Siamese networks and recurrence with attention mechanisms.
Memory-augmented models: Memory-augmented meta-learners train recurrent learners to adapt to new tasks during rollout, including for few-shot recognition and fast reinforcement learning.The experiments report that the proposed method outperforms the recurrent approach on few-shot classification.
Network initialization: Unlike pretrained initialization, the proposed method explicitly optimizes model parameters for fast adaptability to new tasks using only a few examples.The approach can also be viewed as explicitly maximizing new-task loss sensitivity to model parameters.

5. Experimental Evaluation

The evaluation tests whether MAML enables rapid adaptation across regression, few-shot classification, and reinforcement learning. Results show strong few-shot adaptation, continued improvement with additional updates, competitive classification performance, and a substantially cheaper first-order approximation.

Few-shot regression: MAML adapts to sinusoid regression using only 5 datapoints, while standard supervised pretraining fails to adequately adapt.The task varies sinusoid amplitude and phase, and MAML can infer missing curve regions by learning periodic structure.
Few-shot regression: MAML continues improving with additional gradient steps despite being trained for maximal performance after one step.Quantitative results report substantially lower loss than baseline fine-tuning without overfitting to the extremely small meta-test dataset.
Few-shot image recognition: On Omniglot and MiniImagenet few-shot classification, MAML narrowly outperforms prior methods while adding no parameters beyond the classifier weights.The evaluation follows 1- or 5-shot N-way classification protocols on unseen classes.
First-order approximation: The first-order MAML approximation performs nearly the same as full second-order MAML and delivers roughly 33% speed-up in network computation.The approximation omits second derivatives and removes Hessian-vector products from an additional backward pass.
Reinforcement learning: MAML is also evaluated on continuous-control reinforcement-learning tasks using neural-network policies trained with vanilla policy gradient.The task suite includes 2D navigation and goal-direction environments with task-specific goals or velocity objectives.

6. Discussion and Future Work

The method learns easily adaptable model parameters through gradient descent, offering a simple, general-purpose approach applicable across models and learning problems. The authors suggest that reusing knowledge through multitask initialization could support fast training on small datasets and become standard in deep learning and reinforcement learning.

The approach learns model parameters that are easily adaptable through gradient descent.
It is simple and introduces no learned parameters for meta-learning.
The method combines with any model representation amenable to gradient-based training and any differentiable objective, including classification, regression, and reinforcement learning.
Reusing knowledge from past tasks may make high-capacity scalable models, such as deep neural networks, amenable to fast training with small datasets.
The authors envision multitask initialization becoming a standard ingredient in deep learning and reinforcement learning.

A. Additional Experiment Details

This section provides additional details about the experimental setup and hyperparameters.

The section describes additional experimental setup details and hyperparameters.

A.1. Classification

The classification experiments use task-specific gradient-step and step-size configurations, with each gradient computed from NK examples in N-way, K-shot tasks.

A.1. Classification: Each gradient for N-way, K-shot classification is computed using a batch size of NK examples.
A.1. Classification: For Omniglot 5-way models, training uses 1 gradient step with α = 0.4 and meta batch-size 32, while evaluation uses 3 steps with α = 0.4.The 20-way convolutional model uses 5 gradient steps with α = 0.1 for both training and evaluation.

A.2. Reinforcement Learning

The reinforcement-learning experiments used MAML adaptation with a 0.1 learning rate initially and 0.05 thereafter, while baseline step sizes were manually tuned per domain.

A.2. Reinforcement Learning: MAML policies were trained with a single gradient step using α = 0.1.This setting was used in all reinforcement-learning experiments.
A.2. Reinforcement Learning: Halving the learning rate after the first gradient step produced superior evaluation performance.The adaptation schedule therefore used α = 0.1 for the first step and α = 0.05 for subsequent steps.
A.2. Reinforcement Learning: Baseline methods used manually tuned step sizes for each domain.The 2D-navigation experiments used a meta batch size of 20.

B. Additional Sinusoid Results · C. Additional Comparisons

The paper adds quantitative and qualitative sinusoid evaluations of MAML across different shot counts and compares the approach with additional multi-task baselines and Rei (2015).

B. Additional Sinusoid Results: Figure 6 reports full quantitative results for MAML trained on 10-shot learning and evaluated on 5-shot, 10-shot, and 20-shot tasks.The evaluation spans three shot-count settings.
B. Additional Sinusoid Results: MAML’s quantitative sinusoid performance is evaluated after training specifically on 10-shot learning.The training condition is fixed at 10-shot learning.
B. Additional Sinusoid Results: The quantitative evaluation includes 5-shot, 10-shot, and 20-shot testing conditions.These are the three evaluation settings reported in Figure 6.
B. Additional Sinusoid Results: Figure 7 presents qualitative performance comparisons between MAML and a pretrained baseline on randomly sampled sinusoids.The qualitative comparison uses randomly sampled sinusoid tasks.
C. Additional Comparisons: The additional-comparisons section provides more thorough evaluations of the proposed approach.The section expands the evaluation beyond the earlier results.
C. Additional Comparisons: The expanded evaluation includes additional multi-task baselines.These baselines are included as part of the section’s broader comparisons.
C. Additional Comparisons: The paper compares its approach with a method representative of Rei (2015).This comparison is explicitly identified as representative of Rei (2015)’s approach.

C.1. Multi-task baselines

The section compares output-space and parameter-space multi-task baselines for few-shot adaptation. It also evaluates three regularization variants for the parameter-averaging approach.

C.1. Multi-task baselines: The pretraining baseline trained one network across all tasks, then fine-tuned it on each test task using K examples.Because tasks can assign different outputs to the same input, pretraining may learn the average output for that input.
C.1. Multi-task baselines: The parameter-space baseline sequentially trained 500 randomly initialized models on 500 tasks, averaged their parameter vectors, and fine-tuned on 5 datapoints.Each model was trained on a large amount of data from its assigned task, with a tuned step size used during fine-tuning.
C.1. Multi-task baselines: The parameter-averaging setup tested no regularization, standard ℓ2 weight decay, and ℓ2 regularization toward the mean parameter vector of previously trained regressors.The regularized variants encouraged parsimonious solutions, with regularization set as high as possible without significantly deterring performance.

C.2. Context vector adaptation

The context-vector baseline adapts only free input-concatenated parameters rather than MAML’s model parameters. It performs well on toy pointmass reinforcement learning but underperforms on harder problems, likely because its meta-optimization is less flexible.

Method: The comparison learns free context parameters z concatenated with input x, while gradient steps update z instead of the model parameters θ.This follows Rei’s online-adapted context-vector method and applies the same adaptation procedure used for MAML.
Results: MAML continues improving with additional gradient steps on extremely small sinusoid-regression datasets without overfitting, achieving substantially lower loss than baseline fine-tuning.The figure reports test-time learning curves while varying the number of K test-time samples, using the same K examples for each gradient step.
Implementation: The context vector z was concatenated channel-wise with the input image for the Omniglot and reinforcement-learning experiments.The method was evaluated under the same experimental protocol across Omniglot and two reinforcement-learning domains.
Results: The adaptable context vector performed well on the toy pointmass problem but was sub-par on more difficult problems.The authors attribute the weaker results likely to a less flexible meta-optimization.