PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-Learning with Warped Gradient Descent

Sebastian Flennerhag, Andrei A. Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, Raia Hadsell

arXiv:1909.00025v2cs.LGcs.NEstat.ML

TL;DR

Meta-learning seeks efficient update rules for rapidly adapting to new tasks, but direct update generators lack useful inductive bias and gradient-based methods often require backpropagation through adaptation. WarpGrad meta-learns preconditioning through interleaved nonlinear warp-layers without backpropagating through task training, and reports effectiveness across few-shot, supervised, reinforcement, and continual learning. Its objective remains linearly dependent on the number of adaptation steps, leaving scalability of the objective an unresolved limitation.

  • Problem

    Meta-learning lacks an efficient update rule that promotes rapid adaptation while avoiding the brittleness of direct update generators and the trajectory dependence of gradient-based methods.

    Source: §1 · p. 1, §2.1 · p. 3

  • Method

    WarpGrad meta-learns data-dependent gradient preconditioning through nonlinear warp-layers interleaved with task-learner layers, using a trajectory-agnostic objective.

    Source: §1 · p. 2, §2.2 · p. 4, §2.4 · p. 6

  • Results

    WarpGrad outperforms competing methods across few-shot, multi-shot supervised, reinforcement, and continual learning settings while scaling beyond few-shot adaptation.

    Source: §1 · p. 2, §4.2 · p. 9, §5 · p. 10

  • Takeaways & Limitations

    WarpGrad provides a general-purpose meta-learning framework that retains gradient-based inductive bias while supporting complex problems, architectures, and adaptation processes.

    Source: §2.2 · p. 4, §2.4 · p. 6, §5 · p. 10

    The WarpGrad objective samples parameter trajectories and therefore has linear computational complexity in the number of adaptation steps.

    Source: §5 · p. 10

Abstract

from arXiv · show

Learning an efficient update rule from data that promotes rapid learning of new tasks from the same distribution remains an open problem in meta-learning. Typically, previous works have approached this issue either by attempting to train a neural network that directly produces updates or by attempting to learn better initialisations or scaling factors for a gradient-based update rule. Both of these approaches pose challenges. On one hand, directly producing an update forgoes a useful inductive bias and can easily lead to non-converging behaviour. On the other hand, approaches that try to control a gradient-based update rule typically resort to computing gradients through the learning process to obtain their meta-gradients, leading to methods that can not scale beyond few-shot task adaptation. In this work, we propose Warped Gradient Descent (WarpGrad), a method that intersects these approaches to mitigate their limitations. WarpGrad meta-learns an efficiently parameterised preconditioning matrix that facilitates gradient descent across the task distribution. Preconditioning arises by interleaving non-linear layers, referred to as warp-layers, between the layers of a task-learner. Warp-layers are meta-learned without backpropagating through the task training process in a manner similar to methods that learn to directly produce updates. WarpGrad is computationally efficient, easy to implement, and can scale to arbitrarily large meta-learning problems. We provide a geometrical interpretation of the approach and evaluate its effectiveness in a variety of settings, including few-shot, standard supervised, continual and reinforcement learning.

1 INTRODUCTION

WarpGrad combines gradient descent’s inductive bias with flexible, meta-learned preconditioning to address limitations of direct update-rule learning and initialization-based methods. Warp-layers enable this preconditioning across diverse adaptation settings, and experiments report gains over gradient-based baselines while scaling beyond few-shot learning.

  • Direct update-rule methods lack convergence-oriented inductive bias, whereas initialization-based gradient methods restrict knowledge transfer to the initialization.Source: §1 · p. 1
  • Warp-layers interleaved between task-learner layers modulate activations and gradients, producing data-dependent preconditioning beyond prior block-diagonal structures.Source: Figure 1 · p. 2, §1 · p. 2
  • WarpGrad meta-learns a trajectory-agnostic preconditioning rule that combines gradient-based inductive bias with the flexibility of memory-based meta-learners.Source: §1 · p. 2
  • WarpGrad surpasses baseline gradient-based meta-learners on standard few-shot tasks and extends to multi-shot supervised, reinforcement, and continual learning settings.Source: §1 · p. 2

2 WARPED GRADIENT DESCENT

WarpGrad learns a trajectory-agnostic gradient preconditioner through nonlinear warp-layers, combining gradient descent’s inductive bias with meta-learning that avoids backpropagation through task-learning trajectories. Its geometry-based formulation supports flexible adaptation processes and is evaluated across few-shot, supervised, continual, and reinforcement-learning settings.

2.1 GRADIENT-BASED META-LEARNING

  • Backpropagating through gradient-based adaptation makes existing methods computationally expensive and susceptible to exploding or vanishing gradients and credit-assignment problems.These limitations arise from trajectory dependence and restrict such methods to few-shot learning.Source: §2.1 · p. 3
  • WarpGrad defines a trajectory-agnostic meta-objective that learns gradient preconditioning over a joint task-and-parameter search space.The method treats preconditioning as a recurrent operator of order 1 that depends on the current state rather than the full trajectory.Source: §2.1 · p. 3, §2.3 · p. 5, §2.4 · p. 6

2.2 GENERAL-PURPOSE PRECONDITIONING

  • Warp-layers are nonlinear neural networks interleaved with task-learner layers, inducing data-dependent preconditioning through their Jacobians.Nonlinear warp-layers can model preconditioning beyond prior block-diagonal structures and support arbitrary adaptation processes, including recurrent networks.Source: §2.2 · pp. 3–4

2.5 INTEGRATION WITH LEARNED INITIALISATIONS

  • WarpGrad surpasses baseline gradient-based meta-learners on standard few-shot benchmarks, scales to multi-shot supervised settings, and outperforms competing methods in a maze-navigation reinforcement-learning setting.The paper also reports a recurrent warp-layer agent outperforming memory-based meta-learners on a maze task requiring memory, while the synthetic experiment shows smoother warped loss surfaces.Source: §2.2 · p. 4

2.3 THE GEOMETRY OF WARPED GRADIENT DESCENT

  • WarpGrad’s warp-layers define a geometry whose inverse metric is G^-1 := [DxΩ][DxΩ]T, and whose parameter-space update is first-order equivalent to Riemannian descent in the warped task space.The equivalence follows by bounding the error introduced by taking gradient steps in the parameterisation space.Source: §2.3 · p. 5

2.4 META-LEARNING WARP PARAMETERS

  • WarpGrad avoids backpropagation through learning processes, making its meta-gradient independent of the number of task-gradient steps and avoiding trajectory-related vanishing, exploding, and credit-assignment problems by design.The method uses an expectation over sampled gradient-update steps from the task-adaptation search space and requires second-order gradients.Source: §2.3 · p. 5, §2.4 · p. 6

3 RELATED WORK

Related work spans direct update-rule learning, gradient-based adaptation, and preconditioning methods. WarpGrad is positioned as model-embedded preconditioning that uses global information beyond the current task.

  • Early meta-learning work explored evolutionary strategies, recurrent gradient descent, and neural networks that directly predict task-learner parameters.Source: §3 · p. 7
  • Slow and fast weight approaches separate meta-information from rapidly adapting task parameters, a perspective related to WarpGrad’s learned warp-parameters and fast weights.Source: §3 · p. 7
  • Gradient-based few-shot methods extend MAML by sharing parameters that remain fixed during task training but are meta-learned.Source: §3 · p. 7
  • Meta-learned preconditioning connects WarpGrad to second-order optimization, Mirror Descent, and model-embedded preconditioning for high-dimensional non-convex objectives.Source: §3 · p. 8

4 EXPERIMENTS

WarpGrad is evaluated for few-shot retention, scaling beyond few-shot adaptation, and complex meta-learning settings including reinforcement and continual learning. Across these experiments, it improves over relevant baselines and learns sequential tasks with limited forgetting.

Evaluation aims

  • The experiments are designed to test retention of MAML-like inductive bias, scalability beyond few-shot learning, and generalisation to complex meta-learning problems.The evaluation includes few-shot, multi-shot supervised, reinforcement-learning, and continual-learning settings.Source: §4 · p. 8, §4.1 · p. 8, §4.2 · p. 9

Few-shot learning

  • Warp-MAML improves 1- and 5-shot accuracy over all baselines on miniImageNet and tieredImageNet.The gains are 3.6 and 5.5 percentage points on miniImageNet, and 5.2 and 3.8 percentage points on tieredImageNet.Source: §4.1 · p. 8

Multi-shot learning

  • Warp-Leap scales to a 640-step tieredImageNet protocol that MAML-based approaches cannot backpropagate through at the evaluated model size.Warp-Leap surpasses Reptile by 3.88 percentage points, while Reptile outperforms Leap by 2.6 percentage points.Source: §4.2 · p. 9
  • Warp-Leap improves multi-shot Omniglot performance over Leap and Reptile by 8.1 and 12.8 points, respectively.Except with a single meta-training task, it substantially outperforms all baselines and reduces final test error from ~30% to ~15%; non-linear warps reach ~11%.Source: §4.2 · p. 9

Reinforcement learning

  • Warp-RNN reaches ~160 mean cumulative reward in 60,000 episodes, whereas baselines reach at most ~125 after 100,000 episodes.Linear warps perform worse than the baseline RNN on this maze-navigation task.Source: §4.3 · p. 10

Continual learning

  • WarpGrad learns new continual-learning sub-tasks with mean losses on the order of 10^-3, followed by stable performance near ~10^-2 after switches.The experiment uses 100 random tasks, each containing five sub-tasks, and evaluates both sequential and random sub-task orders.Source: §4.3 · p. 10, Figure 5 · p. 10

5 CONCLUSION

WarpGrad combines gradient-based inductive bias with the flexibility of memory-based meta-learners by meta-learning gradient preconditioning without backpropagating through adaptation. The framework supports multiple architecture designs and remains limited by trajectory-sampling cost that grows linearly with adaptation steps.

  • WarpGrad combines memory-based flexibility with gradient-based inductive bias while preconditioning gradients without backpropagating through task adaptation.Source: §5 · p. 10
  • Warp-layers are universal function approximators that let WarpGrad express geometries beyond prior block-diagonal preconditioning structures.Source: §5 · p. 10
  • The current WarpGrad objective samples parameter trajectories, giving it linear computational complexity in the number of adaptation steps.Source: §5 · p. 10
  • WarpGrad can embed preconditioning by inserting warp-layers or designating existing layers as task-adaptable and warp layers.Source: A WARPGRAD DESIGN PRINCIPLES FOR… · p. 15, Figure 6 · p. 15
  • The framework is compatible with architectures including residual networks and LSTMs because standard backpropagation automatically warps gradients once warp-layers are chosen.Source: A WARPGRAD DESIGN PRINCIPLES FOR… · pp. 15–16

B WARPGRAD META-TRAINING ALGORITHMS

WarpGrad provides online, offline, and continual meta-training algorithms for accumulating and applying meta-gradients across task adaptation. Offline replay-buffer training can improve data efficiency, while continual training updates warp parameters during streams of learning experiences.

Training algorithms

WarpGrad optimisers

  • Warp-MAML applies the full warp objective together with the MAML objective for few-shot learning, whereas Warp-Leap applies Leap to the task-parameter initialisation for multi-shot learning.Source: C WARPGRAD OPTIMISERS · p. 16

Experimental variants

  • The paper compares exact and approximate meta-objectives and several warp-layer implementations in an Omniglot ablation study.Source: C WARPGRAD OPTIMISERS · p. 17

WarpGrad optimisers

  • The reinforcement-learning Warp-RNN meta-learns an LSTM that modulates task-learner weights using continual meta-training over episode streams.Source: C WARPGRAD OPTIMISERS · p. 17

D SYNTHETIC EXPERIMENT

The synthetic experiment tests whether WarpGrad can learn a geometry that improves optimization across randomly generated but structurally related two-dimensional loss surfaces. From difficult shared initialisations, the learned warp reshapes the optimization geometry for smoother convergence.

Task construction

  • The synthetic tasks minimize randomly generated two-dimensional objectives whose loss surfaces differ but share an underlying structure.Source: D SYNTHETIC EXPERIMENT · p. 18
  • Each task is optimized for 100 steps from a random initialisation with learning rate 0.1 during meta-training.Source: D SYNTHETIC EXPERIMENT · p. 18

Results

  • Figure 7 compares Gradient Descent and WarpGrad from the same initialisation across three task loss surfaces, showing representation-space and model-parameter trajectories.Source: Figure 7 · p. 19
  • WarpGrad learns a geometry that is robust to difficult initialisations and smoothly warps the native loss surface.Source: D SYNTHETIC EXPERIMENT · p. 19

E OMNIGLOT

The Omniglot experiment evaluates Warp-Leap on held-out alphabet tasks using convolutional warp-layers inserted between convolutional blocks. Offline replay-buffer mini-batching substantially accelerates meta-training, while the benchmark compares Warp-Leap with gradient-based and other meta-learning baselines.

Dataset and setup

  • Omniglot contains 46 retained alphabet tasks, with 10 alphabets held out for final meta-testing and each task posed as 20-way classification.Source: E OMNIGLOT · p. 19

Model

  • Warp-Leap inserts convolutional warp-layers between convolutional blocks and finds that simple warp-layers perform well, with added capacity improving generalisation.Source: E OMNIGLOT · p. 19

Evaluation

Training

  • Offline training collects 2000 task parameterisations into a replay buffer before repeatedly updating warp parameters from sampled mini-batches.Source: E OMNIGLOT · p. 20
  • Replay-buffer mini-batching makes meta-training converge much faster without discernible instability.Source: E OMNIGLOT · p. 20

F ABLATION STUDY: WARP LAYERS, META-OBJECTIVE, AND META-TRAINING

The ablation study varies meta-training objectives, online versus offline protocols, learning-rate meta-learning, and warp-layer architectures to assess WarpGrad design choices.

Evaluation protocol

  • Warp-Leap evaluates these alternatives with 25 pretraining tasks and reports final test accuracy over four independent runs.All ablations share hyper-parameters except online meta-training, which uses a different learning rate.Source: F ABLATION STUDY: WARP LAYERS,… · p. 21

Meta-objective and meta-training

  • The study compares approximate and full meta-objectives, online and offline meta-training, and whether to meta-learn a scalar task-adaptation learning rate.Warp parameters can learn layer-wise scaling; online meta-training uses a distinct learning rate.Source: F ABLATION STUDY: WARP LAYERS,… · p. 21

Warp-layer architecture

  • Warp-layer architectures range from channelwise scaling to nonlinear residual designs, two stacked convolutions, and a FiLM-equipped two-layer architecture.The convolutional variants use 128 filters in the first warp convolution and 64 in the second.Source: F ABLATION STUDY: WARP LAYERS,… · p. 21

G ABLATION STUDY: WARPGRAD AND NATURAL GRADIENT DESCENT

This ablation compares WarpGrad with SGD and KFAC and examines whether its learned geometry resembles the Fisher geometry. WarpGrad improves adaptation beyond block-diagonal preconditioning, but its learned covariance structure differs from the Fisher expectation.

Geometry analysis

WarpGrad versus gradient baselines

  • WarpGrad’s warped geometries facilitate held-out-task adaptation more than SGD or KFAC, with performance improving further beyond block-diagonal preconditioning.The comparison fixes a random initialization and meta-learns only warp parameters.Source: G ABLATION STUDY: WARPGRAD AND NATURAL… · pp. 21–22

Geometry analysis

  • WarpGrad-Leap generally produces zero-centered post-warp activations, while post-warp covariances remain dissimilar from identity and vary across layers.Positive pre-warp activations are attributed to ReLU.Source: G ABLATION STUDY: WARPGRAD AND NATURAL… · p. 23

Relation to natural gradients

  • The learned correlation structure differs significantly from the Fisher-matrix expectation, indicating behavior distinct from Natural Gradient Descent.The authors suggest WarpGrad may encode a different geometry using global information beyond the task at hand.Source: G ABLATION STUDY: WARPGRAD AND NATURAL… · p. 23

H miniIMAGENET AND tieredIMAGENET

The image-classification experiments evaluate standard few-shot and multi-shot task settings on miniImageNet and tieredImageNet using held-out meta-test tasks. WarpGrad outperforms the compared methods in multi-shot tieredImageNet learning speed and final performance.

Datasets

  • miniImageNet contains 100 classes split into 64 meta-training, 16 meta-validation, and 20 meta-test classes.Each class has 600 images.Source: H miniIMAGENET AND tieredIMAGENET · p. 23
  • tieredImageNet partitions 608 classes into 34 higher-level categories, using 20 categories for meta-training, 6 for validation, and 8 for testing.The hierarchical split increases separation between meta-training and meta-evaluation.Source: H miniIMAGENET AND tieredIMAGENET · p. 23

Experimental settings

  • Few-shot tasks use 5-way 1-shot or 5-shot episodes, with 15 validation images per class and 600 meta-validation or meta-test instances.Meta-training uses corresponding 5-way 5-shot or 15-shot settings.Source: H miniIMAGENET AND tieredIMAGENET · p. 23
  • Multi-shot experiments use 10-way, 640-shot tasks with 50 validation images per class and six convolutional blocks.Convolutional widths are tuned from {64, 64, 160, 160, 256, 256}.Source: H miniIMAGENET AND tieredIMAGENET · p. 24

Results

  • WarpGrad outperforms Leap, Reptile, and training from scratch in both convergence rate and final test performance on multi-shot tieredImageNet.The figure reports mean learning curves on held-out meta-test tasks and performance during meta-training.Source: H miniIMAGENET AND tieredIMAGENET · p. 24, Figure 10 · p. 24

I MAZE NAVIGATION

The maze-navigation experiments apply WarpGrad to recurrent online meta-learning and reinforcement learning. Warp-RNN converges faster and achieves higher cumulative reward than the baselines while retaining the baseline task-learner size and hidden-state size.

Maze-navigation task

  • Each maze task randomly selects a goal location, which remains fixed during a 200-step episode until the agent reaches it and is teleported.The task therefore requires repeated goal-directed navigation within one episode.Source: I MAZE NAVIGATION · p. 25

Warp-RNN method

  • Warp-RNN uses a frozen-for-most-training LSTM warp-layer to modulate the task RNN, forming a HyperNetwork-style recurrent preconditioner.The task-adaptable parameters remain those of the baseline RNN.Source: I MAZE NAVIGATION · p. 25

Ablations and comparisons

  • Updating the meta-LSTM at every task adaptation performs worse than WarpGrad-RNN, while linear T-Nets-RNN provides a comparison to nonlinear preconditioning.These controls assess meta-LSTM update frequency and the form of preconditioning.Source: I MAZE NAVIGATION · p. 25

Warp-RNN method

  • Warp-RNN meta-training continuously accumulates meta-gradients and updates warp parameters every 30th task-parameter update.The approximate objective is used for the continuous stream of episode batches.Source: I MAZE NAVIGATION · p. 25

Results

  • Warp-RNN converges faster and achieves higher cumulative reward than all baselines with the same task-learner parameter count and hidden-state size.The implementation adds four lines of code to a standard training script.Source: I MAZE NAVIGATION · p. 25

J META-LEARNING FOR CONTINUAL LEARNING

This section evaluates WarpGrad as an optimiser for continual learning, where online updates must learn changing sub-tasks while retaining earlier performance without storing past data. The experiment uses continual sine regression and a meta-objective that averages losses across current and preceding sub-tasks.

  • Online SGD often adapts to the most recently encountered data distribution, producing catastrophic forgetting.WarpGrad is investigated as a way to avoid this problem in fully online learning without retaining past data.Source: J META-LEARNING FOR CONTINUAL LEARNING · pp. 25–26

Continual Sine Regression

  • Continual sine regression splits [−5, 5] into five consecutive sub-tasks presented sequentially, with 20 gradient steps and batches of five samples per sub-task.The full sequence therefore uses K = 100 online adaptation steps, with each update based only on data from the current sub-task.Source: J META-LEARNING FOR CONTINUAL LEARNING · pp. 26–27

Model

  • The task-learner is a four-layer ReLU network augmented with residual warping blocks whose parameters are meta-learned and fixed during adaptation.Each warping block is a two-layer network with 100 hidden units and tanh nonlinearities.Source: J META-LEARNING FOR CONTINUAL LEARNING · pp. 26–27

Meta-learning an optimiser for continual learning

  • The continual-learning meta-objective averages validation losses for the current and every preceding sub-task, giving equal weight to tasks that should be learned or remembered.Losses from earlier sub-tasks recur in the objective across later sequence positions, encouraging joint performance over the sequence.Source: J META-LEARNING FOR CONTINUAL LEARNING · p. 27

Evaluation

Loading 1909.00025v2…