PaperlayerView exact revision on arXiv

Source-linked AI summary

Online Meta-Learning

Chelsea Finn, Aravind Rajeswaran, Sham Kakade, Sergey Levine

arXiv:1902.08438v4cs.LGcs.AIstat.ML

TL;DR

The paper addresses the gap between batch meta-learning and sequential online learning by introducing online meta-learning for continual task streams. It proposes FTML, an online extension of MAML, and reports logarithmic regret, strong empirical performance, and limitations concerning memory and computational efficiency.

  • Problem

    Existing meta-learning assumes tasks are available together or follow fixed distributions, while online learning processes sequential tasks without task-specific adaptation.

  • Method

    The paper formulates online meta-learning and proposes FTML, which extends MAML by combining sequential learning with task-specific adaptation.

  • Results

    The analysis gives FTML an O(log T) regret guarantee, and experiments report that it outperforms prior methods on sequential vision tasks.

  • Takeaways & Limitations

    Online meta-learning offers a framework for using streaming experience to improve current-task learning and future adaptation.

  • Takeaways & Limitations

    The study maintains a buffer of all observed data and leaves limited-memory forgetting and more computationally efficient FTML variants for future work.

Abstract

from arXiv · show

A central capability of intelligent systems is the ability to continuously build upon previous experiences to speed up and enhance learning of new tasks. Two distinct research paradigms have studied this question. Meta-learning views this problem as learning a prior over model parameters that is amenable for fast adaptation on a new task, but typically assumes the set of tasks are available together as a batch. In contrast, online (regret based) learning considers a sequential setting in which problems are revealed one after the other, but conventionally train only a single model without any task-specific adaptation. This work introduces an online meta-learning setting, which merges ideas from both the aforementioned paradigms to better capture the spirit and practice of continual lifelong learning. We propose the follow the meta leader algorithm which extends the MAML algorithm to this setting. Theoretically, this work provides an $\mathcal{O}(\log T)$ regret guarantee with only one additional higher order smoothness assumption in comparison to the standard online setting. Our experimental evaluation on three different large-scale tasks suggest that the proposed algorithm significantly outperforms alternatives based on traditional online learning approaches.

1. Introduction

Online meta-learning combines sequential task processing with task-specific adaptation, addressing limitations of batch meta-learning and conventional online learning for continual learning. The paper introduces FTML, extends MAML to this setting, and reports logarithmic regret and strong empirical performance.

  • Meta-learning uses prior tasks to learn parameters or learning procedures, while online learning processes sequential tasks without task-specific adaptation.
  • Neither paradigm alone captures continual lifelong learning because meta-learning neglects sequential non-stationarity, whereas online learning does not use past experience to accelerate adaptation.
  • Online meta-learning uses sequential experience to learn useful priors while adapting quickly to the current task.
  • Task structure can prevent misleading joint-training conclusions, such as treating sinusoid data as y = 0 or interpreting background color as the MNIST digit label.
  • FTML extends MAML to online meta-learning, achieves O(log T) regret against the best meta-learner in hindsight, and significantly outperforms prior methods on large-scale vision tasks.The experiments use MNIST, CIFAR-100, and PASCAL 3D+ datasets.

2. Foundations

The foundations contrast batch meta-learning, which learns fast adaptation from task families, with online learning, which handles sequential losses and evaluates regret against comparator classes. This contrast motivates an adaptive sequential formulation.

  • Few-shot supervised learning seeks to minimize risk for a task using only a small labeled dataset, while exploiting experience across multiple tasks.
  • Meta-learning assumes tasks come from a fixed distribution and learns an initialization that can be quickly adapted to a new task.
  • MAML learns initial parameters whose few gradient steps on task data minimize the new task’s loss.
  • MAML is not directly suited to sequential, non-stationary settings because it separates meta-training from deployment and generally assumes a fixed task distribution.
  • Online learning processes a sequence of losses that need not follow a fixed distribution and measures performance through regret against a comparator class.
  • Standard FTL consolidates prior task data into one model, whereas the paper seeks a more adaptive comparator because joint training may be ineffective.

3. The Online Meta-Learning Problem

Online meta-learning presents tasks sequentially while allowing local task-specific updates before evaluation. Its objective is to perform well over time against a powerful meta-learned comparator.

  • Each task corresponds to a round, and the learner selects parameters intended to perform well on that round’s loss.
  • An update procedure maps the selected parameters to adapted parameters that perform better on the current task.
  • A gradient step, Ut(w) = w − α∇ˆft(w), is one example of the task-specific update procedure.
  • The round protocol selects wt, reveals ft, applies Ut, incurs ft(˜wt), and advances to the next task.
  • Regret is measured against the best meta-learned model in hindsight, whose comparator can adapt locally to each task.

4. Algorithm and Analysis

FTML adapts follow-the-leader to meta-learning by selecting parameters using accumulated post-update losses. Under smoothness and strong-convexity assumptions, the transformed objectives inherit properties that yield logarithmic regret.

  • 4.1. Follow the Meta Leader: FTML updates parameters by minimizing the accumulated losses after task-specific update procedures, paralleling follow-the-leader.
  • 4.1. Follow the Meta Leader: The FTML choice represents the best meta-learner in hindsight if learning stopped at the current round.
  • 4.2. Assumptions: The analysis assumes bounded gradients, Lipschitz gradients, Lipschitz Hessians, and strong convexity of the loss functions.
  • 4.2. Assumptions: The Lipschitz-Hessian condition is the additional higher-order smoothness assumption used to characterize the MAML-like objective containing a gradient update.
  • 4.3. Analysis: With an appropriately sized one-step gradient update, the transformed objective is convex, 9β/8-smooth, and µ/8-strongly convex.
  • 4.3. Analysis: The convexity result also provides provable and efficient optimization of MAML-like objectives using standard gradient computation.
  • 4.3. Analysis: The inherited strong convexity makes FTML equivalent to FTL on transformed losses and gives an O(log T) regret guarantee.

5. Practical Online Meta-Learning Algorithm

The practical FTML algorithm adapts online meta-learning to high-capacity neural networks using stochastic optimization over previously seen tasks. It incrementally updates a task buffer and evaluates task-specific adaptation as data arrives.

  • Optimization: FTML uses stochastic gradient optimization with training and validation minibatches to adapt the MAML-style objective in high-capacity neural networks.Independent minibatches reduce interaction between inner adaptation and outer optimization, which uses Adam; multiple inner-loop steps can be beneficial.
  • Online procedure: At each round, FTML adds the new task to a buffer and initializes its task-specific dataset before processing incoming data.The dataset grows incrementally as new batches arrive.
  • Online procedure: Each incoming batch triggers a meta-update using data from all tasks seen so far, followed by an update of the current model using the data collected for that task.The meta-update uses fixed-size minibatches, whereas evaluation can use hundreds of datapoints.
  • Evaluation: Task-learning efficiency is measured by the amount of current-task data required to reach a specified proficiency threshold.Reaching the threshold with less data indicates positive transfer from previously learned priors.

6. Experimental Evaluation

The experiments evaluate practical FTML on transformed MNIST, CIFAR-100 classification, and synthetic 3D pose prediction against online-learning and non-meta-learning baselines. Across these settings, FTML generally improves task-learning efficiency, with task- and data-dependent differences in final performance and transfer.

  • Experimental setup: FTML is evaluated on synthetic MNIST transformations, sequential CIFAR-100 classification, and synthetic PASCAL3D+ pose prediction.The study compares practical FTML with train-on-everything, training-from-scratch, joint training with fine-tuning, and related variants.
  • Rainbow MNIST: 56 Rainbow MNIST tasks vary background color, scale, and rotation, with 900 images per task and a 90% accuracy proficiency threshold.Tasks are presented in random order.
  • Rainbow MNIST: FTML learns Rainbow MNIST tasks increasingly quickly and substantially outperforms alternatives in both efficiency and final performance.Independent models reach similar final performance after 900 datapoints but learn less efficiently.
  • Sequential Object Pose Prediction: In pose prediction, meta-learning improves both learning efficiency and performance, solving many tasks with only 10 datapoints.TOE outperforms training from scratch in this structurally similar task family, but FTML performs better than TOE; FTL is comparable or worse than TOE.
  • Five-Way CIFAR-100: On CIFAR-100, FTML learns more efficiently than models trained from scratch and shared-feature variants, while training from scratch reaches similar performance after 2000 datapoints.Adapting all layers benefits FTML relative to adapting only the final layer.

7. Connections to Related Work

The paper situates online meta-learning among meta-learning, continual learning, and online learning, distinguishing its adaptive comparator and forward-transfer focus. It presents FTML as a simple MAML extension with theoretical guarantees in sequential, potentially non-stationary settings.

  • Meta-learning: Prior meta-learning work rarely evaluated algorithms on continuous task streams, whereas this work adapts all model parameters for each task and provides theoretical guarantees.The approach differs from methods that adapt only hyperparameters or use mixtures over meta-learned parameters.
  • Continual learning: The paper focuses on forward transfer—improving the efficiency of learning new tasks—rather than catastrophic forgetting.It sidesteps forgetting by maintaining a buffer containing all observed data.
  • Online meta-learning: Online meta-learning combines sequential task exposure from online learning with task-specific adaptation from meta-learning.Unlike conventional online learning, its comparator can perform task-specific updates; unlike typical meta-learning, tasks need not be available jointly upfront.
  • Online meta-learning: FTML extends MAML to online learning and is analogous to follow the leader.The algorithm is designed for streaming tasks and competes with a powerful adaptive comparator class.
  • Online learning: FTML obtains sublinear regret against a powerful comparator that can produce different models for different loss functions.The paper contrasts this with fixed-model hindsight comparisons and reports that related algorithms lacked large-scale, non-stationary empirical evaluation.

8. Discussion and Future Work

The discussion presents online meta-learning as a framework connecting meta-learning and online learning for continually changing environments. It reports logarithmic regret and empirical gains while identifying update-rule, memory, computational, and scalability challenges.

  • Discussion: Online meta-learning aims to let an agent use streaming experience to master current tasks and become more proficient at learning future tasks.The paper frames this as a perspective on learning in a constantly changing environment.
  • Discussion: FTML achieves logarithmic regret, has a practical adaptation, and outperforms prior methods in the experimental evaluation.These are the paper’s stated theoretical, practical, and empirical outcomes.
  • Future work: The analysis concentrates on one gradient-descent update, although the experiments use multiple gradient steps and backpropagation through the full update path.Analyzing multiple-step and higher-order update rules remains future work.
  • Future work: FTML’s computational cost grows as losses accumulate, and practical settings may make storing all previous datapoints difficult.The method learned nearly 100 sequential tasks without significant compute or memory burdens, but the authors state that scalability remains a concern.

A. Linear Regression Example

The linear regression example shows that joint training and MAML can produce different solutions, with meta-learning improving post-adaptation performance even for quadratic objectives.

  • Problem setup: The example studies a collection of quadratic objectives representing linear regression tasks with task-specific input-output distributions.Each task uses a linear predictor, with quadratic coefficients determined by moments of its data distribution.
  • Optimization comparison: Joint training and MAML optimize different objectives for these tasks, yielding distinct solutions.The analysis explicitly derives separate solutions for joint training and the MAML objective.
  • Optimization comparison: The difference between the solutions is attributed to differences in moments of the input distributions.The MAML solution uses a weighted average whose weights depend on the task-specific matrices A_i.
  • Performance implication: Optimizing the MAML objective explicitly would lead to better performance after task-specific adaptation.This connects the objective mismatch to the performance evaluated after adaptation.
  • Performance implication: Meta-learning can outperform joint training even with quadratic loss functions and simple convex loss landscapes.The example provides a theoretical separation complementary to empirical gains observed with neural networks.

B. Theoretical Analysis

The theoretical analysis develops norm and gradient-descent tools, including a contraction result under Lipschitz, smoothness, and strong-convexity conditions.

  • Preliminaries: The analysis begins by reviewing vector and matrix norms, positive semidefinite ordering, convexity, Lipschitz continuity, and smoothness.These definitions establish the notation and inequalities used in later proofs.
  • Assumptions: The paper defines G-Lipschitz, β-smooth, ρ-Hessian-Lipschitz, and μ-strongly convex functions for its analysis.The Hessian-Lipschitz condition captures higher-order smoothness beyond standard online-learning assumptions.
  • Analytical tools: A line-integral argument and mean-value inequality provide bounds for differences between function values and gradients.The proof uses Jacobian norms together with Cauchy-Schwarz and sub-multiplicativity.
  • Gradient-descent dynamics: Gradient descent is contractive for a G-Lipschitz, β-smooth, μ-strongly convex function when the step size satisfies the stated bound.The contraction follows by bounding the update Jacobian using Hessian eigenvalue bounds.

B.3. Main Theorem

The main theorem shows that one-step adaptation preserves useful convexity and smoothness properties under standard assumptions plus a Hessian-Lipschitz condition.

  • Assumptions: The theorem assumes twice differentiability, function-value Lipschitzness, β-smoothness, ρ-Lipschitz Hessians, and μ-strong convexity.The higher-order Hessian condition is the additional smoothness requirement used to analyze the adapted objective.
  • Adapted objective: The adapted function ˜f is defined by evaluating f after a one-step gradient update involving ˆf.This embeds the adaptation procedure into the objective analyzed by the theorem.
  • Theorem guarantee: For the stated step-size condition, ˜f is convex, 9β/8-smooth, and μ/8-strongly convex.The theorem transfers these properties to the function evaluated after one gradient update, with modified constants.
  • Proof strategy: The proof bounds changes in the adapted gradient using the chain rule, norm inequalities, Hessian Lipschitzness, and gradient-descent contraction.The argument separately derives upper and lower bounds before establishing the resulting curvature properties.
  • Proof strategy: The lower-bound argument combines previously established terms with strong convexity and smoothness bounds.These bounds support the strong-convexity conclusion for the adapted function.

C. Additional Experimental Details

The experiments use multi-step inner adaptation, task-dependent batch sizes, and convolutional architectures across MNIST, CIFAR, and pose prediction settings.

  • Optimization: FTML uses 5 inner batch gradient-descent steps with step size α = 0.1 in all experiments.Inner batches contain 10 examples for MNIST and pose prediction and 25 datapoints for CIFAR.
  • Optimization: The experiments use Adam for convolutional networks except CIFAR NML, where SGD with momentum is used because Adam was unstable.The CIFAR NML momentum parameter is 0.9.
  • Losses: MNIST and CIFAR use cross-entropy loss with label smoothing ε = 0.1, including for the FTML inner loss.The same loss formulation is used for the inner FTML objective.
  • Architectures: MNIST and CIFAR use five-layer convolutional networks, while pose prediction uses four convolutional layers with spatial soft-argmax processing.The architectures differ across image classification and pose prediction experiments.
Loading 1902.08438v4…