PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-Learning with Warped Gradient Descent

Sebastian Flennerhag, Andrei A. Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, Raia Hadsell

arXiv:1909.00025v2cs.LGcs.NEstat.ML

TL;DR

Meta-learning seeks efficient update rules for rapidly adapting to new tasks, but direct update generators lack useful inductive bias and gradient-based methods often require backpropagation through adaptation. WarpGrad meta-learns preconditioning through interleaved nonlinear warp-layers without backpropagating through task training, and reports effectiveness across few-shot, supervised, reinforcement, and continual learning. Its objective remains linearly dependent on the number of adaptation steps, leaving scalability of the objective an unresolved limitation.

  • Problem

    Meta-learning lacks an efficient update rule that promotes rapid adaptation while avoiding the brittleness of direct update generators and the trajectory dependence of gradient-based methods.

  • Method

    WarpGrad meta-learns data-dependent gradient preconditioning through nonlinear warp-layers interleaved with task-learner layers, using a trajectory-agnostic objective.

  • Results

    WarpGrad outperforms competing methods across few-shot, multi-shot supervised, reinforcement, and continual learning settings while scaling beyond few-shot adaptation.

  • Takeaways & Limitations

    WarpGrad provides a general-purpose meta-learning framework that retains gradient-based inductive bias while supporting complex problems, architectures, and adaptation processes.

  • Takeaways & Limitations

    The WarpGrad objective samples parameter trajectories and therefore has linear computational complexity in the number of adaptation steps.

Abstract

from arXiv · show

Learning an efficient update rule from data that promotes rapid learning of new tasks from the same distribution remains an open problem in meta-learning. Typically, previous works have approached this issue either by attempting to train a neural network that directly produces updates or by attempting to learn better initialisations or scaling factors for a gradient-based update rule. Both of these approaches pose challenges. On one hand, directly producing an update forgoes a useful inductive bias and can easily lead to non-converging behaviour. On the other hand, approaches that try to control a gradient-based update rule typically resort to computing gradients through the learning process to obtain their meta-gradients, leading to methods that can not scale beyond few-shot task adaptation. In this work, we propose Warped Gradient Descent (WarpGrad), a method that intersects these approaches to mitigate their limitations. WarpGrad meta-learns an efficiently parameterised preconditioning matrix that facilitates gradient descent across the task distribution. Preconditioning arises by interleaving non-linear layers, referred to as warp-layers, between the layers of a task-learner. Warp-layers are meta-learned without backpropagating through the task training process in a manner similar to methods that learn to directly produce updates. WarpGrad is computationally efficient, easy to implement, and can scale to arbitrarily large meta-learning problems. We provide a geometrical interpretation of the approach and evaluate its effectiveness in a variety of settings, including few-shot, standard supervised, continual and reinforcement learning.

1 INTRODUCTION

WarpGrad combines gradient descent’s inductive bias with flexible, meta-learned preconditioning to address limitations of direct update-rule learning and initialization-based methods. Warp-layers enable this preconditioning across diverse adaptation settings, and experiments report gains over gradient-based baselines while scaling beyond few-shot learning.

  • Direct update-rule methods lack convergence-oriented inductive bias, whereas initialization-based gradient methods restrict knowledge transfer to the initialization.
  • Warp-layers interleaved between task-learner layers modulate activations and gradients, producing data-dependent preconditioning beyond prior block-diagonal structures.
  • WarpGrad meta-learns a trajectory-agnostic preconditioning rule that combines gradient-based inductive bias with the flexibility of memory-based meta-learners.
  • WarpGrad surpasses baseline gradient-based meta-learners on standard few-shot tasks and extends to multi-shot supervised, reinforcement, and continual learning settings.

2 WARPED GRADIENT DESCENT

WarpGrad learns a trajectory-agnostic gradient preconditioner through nonlinear warp-layers, combining gradient descent’s inductive bias with meta-learning that avoids backpropagation through task-learning trajectories. Its geometry-based formulation supports flexible adaptation processes and is evaluated across few-shot, supervised, continual, and reinforcement-learning settings.

  • 2.1 GRADIENT-BASED META-LEARNING: Backpropagating through gradient-based adaptation makes existing methods computationally expensive and susceptible to exploding or vanishing gradients and credit-assignment problems.These limitations arise from trajectory dependence and restrict such methods to few-shot learning.
  • 2.1 GRADIENT-BASED META-LEARNING: WarpGrad defines a trajectory-agnostic meta-objective that learns gradient preconditioning over a joint task-and-parameter search space.The method treats preconditioning as a recurrent operator of order 1 that depends on the current state rather than the full trajectory.
  • 2.2 GENERAL-PURPOSE PRECONDITIONING: Warp-layers are nonlinear neural networks interleaved with task-learner layers, inducing data-dependent preconditioning through their Jacobians.Nonlinear warp-layers can model preconditioning beyond prior block-diagonal structures and support arbitrary adaptation processes, including recurrent networks.
  • 2.5 INTEGRATION WITH LEARNED INITIALISATIONS: WarpGrad surpasses baseline gradient-based meta-learners on standard few-shot benchmarks, scales to multi-shot supervised settings, and outperforms competing methods in a maze-navigation reinforcement-learning setting.The paper also reports a recurrent warp-layer agent outperforming memory-based meta-learners on a maze task requiring memory, while the synthetic experiment shows smoother warped loss surfaces.
  • 2.3 THE GEOMETRY OF WARPED GRADIENT DESCENT: WarpGrad’s warp-layers define a geometry whose inverse metric is G^-1 := [DxΩ][DxΩ]T, and whose parameter-space update is first-order equivalent to Riemannian descent in the warped task space.The equivalence follows by bounding the error introduced by taking gradient steps in the parameterisation space.
  • 2.4 META-LEARNING WARP PARAMETERS: WarpGrad avoids backpropagation through learning processes, making its meta-gradient independent of the number of task-gradient steps and avoiding trajectory-related vanishing, exploding, and credit-assignment problems by design.The method uses an expectation over sampled gradient-update steps from the task-adaptation search space and requires second-order gradients.

3 RELATED WORK

Related work spans direct update-rule learning, gradient-based adaptation, and preconditioning methods. WarpGrad is positioned as model-embedded preconditioning that uses global information beyond the current task.

  • Early meta-learning work explored evolutionary strategies, recurrent gradient descent, and neural networks that directly predict task-learner parameters.
  • Slow and fast weight approaches separate meta-information from rapidly adapting task parameters, a perspective related to WarpGrad’s learned warp-parameters and fast weights.
  • Gradient-based few-shot methods extend MAML by sharing parameters that remain fixed during task training but are meta-learned.
  • Meta-learned preconditioning connects WarpGrad to second-order optimization, Mirror Descent, and model-embedded preconditioning for high-dimensional non-convex objectives.

4 EXPERIMENTS

WarpGrad is evaluated for few-shot retention, scaling beyond few-shot adaptation, and complex meta-learning settings including reinforcement and continual learning. Across these experiments, it improves over relevant baselines and learns sequential tasks with limited forgetting.

  • Evaluation aims: The experiments are designed to test retention of MAML-like inductive bias, scalability beyond few-shot learning, and generalisation to complex meta-learning problems.The evaluation includes few-shot, multi-shot supervised, reinforcement-learning, and continual-learning settings.
  • Few-shot learning: Warp-MAML improves 1- and 5-shot accuracy over all baselines on miniImageNet and tieredImageNet.The gains are 3.6 and 5.5 percentage points on miniImageNet, and 5.2 and 3.8 percentage points on tieredImageNet.
  • Multi-shot learning: Warp-Leap scales to a 640-step tieredImageNet protocol that MAML-based approaches cannot backpropagate through at the evaluated model size.Warp-Leap surpasses Reptile by 3.88 percentage points, while Reptile outperforms Leap by 2.6 percentage points.
  • Multi-shot learning: Warp-Leap improves multi-shot Omniglot performance over Leap and Reptile by 8.1 and 12.8 points, respectively.Except with a single meta-training task, it substantially outperforms all baselines and reduces final test error from ~30% to ~15%; non-linear warps reach ~11%.
  • Reinforcement learning: Warp-RNN reaches ~160 mean cumulative reward in 60,000 episodes, whereas baselines reach at most ~125 after 100,000 episodes.Linear warps perform worse than the baseline RNN on this maze-navigation task.
  • Continual learning: WarpGrad learns new continual-learning sub-tasks with mean losses on the order of 10^-3, followed by stable performance near ~10^-2 after switches.The experiment uses 100 random tasks, each containing five sub-tasks, and evaluates both sequential and random sub-task orders.

5 CONCLUSION

WarpGrad combines gradient-based inductive bias with the flexibility of memory-based meta-learners by meta-learning gradient preconditioning without backpropagating through adaptation. The framework supports multiple architecture designs and remains limited by trajectory-sampling cost that grows linearly with adaptation steps.

  • WarpGrad combines memory-based flexibility with gradient-based inductive bias while preconditioning gradients without backpropagating through task adaptation.
  • Warp-layers are universal function approximators that let WarpGrad express geometries beyond prior block-diagonal preconditioning structures.
  • The current WarpGrad objective samples parameter trajectories, giving it linear computational complexity in the number of adaptation steps.
  • WarpGrad can embed preconditioning by inserting warp-layers or designating existing layers as task-adaptable and warp layers.
  • The framework is compatible with architectures including residual networks and LSTMs because standard backpropagation automatically warps gradients once warp-layers are chosen.

B WARPGRAD META-TRAINING ALGORITHMS

WarpGrad provides online, offline, and continual meta-training algorithms for accumulating and applying meta-gradients across task adaptation. Offline replay-buffer training can improve data efficiency, while continual training updates warp parameters during streams of learning experiences.

  • Training algorithms: Online WarpGrad meta-training accumulates meta-gradients during task adaptation with constant memory and linear scaling in trajectory length.
  • Training algorithms: Offline meta-training stores sampled task parameterisations in a replay buffer, enabling mini-batching and multiple warp-parameter updates per trajectory sample.
  • Training algorithms: Continual meta-training accumulates meta-gradients across a stream of task experiences and updates warp parameters intermittently.
  • WarpGrad optimisers: Warp-MAML applies the full warp objective together with the MAML objective for few-shot learning, whereas Warp-Leap applies Leap to the task-parameter initialisation for multi-shot learning.
  • Experimental variants: The paper compares exact and approximate meta-objectives and several warp-layer implementations in an Omniglot ablation study.
  • WarpGrad optimisers: The reinforcement-learning Warp-RNN meta-learns an LSTM that modulates task-learner weights using continual meta-training over episode streams.

D SYNTHETIC EXPERIMENT

The synthetic experiment tests whether WarpGrad can learn a geometry that improves optimization across randomly generated but structurally related two-dimensional loss surfaces. From difficult shared initialisations, the learned warp reshapes the optimization geometry for smoother convergence.

  • Task construction: The synthetic tasks minimize randomly generated two-dimensional objectives whose loss surfaces differ but share an underlying structure.
  • Task construction: Each task is optimized for 100 steps from a random initialisation with learning rate 0.1 during meta-training.
  • Results: Figure 7 compares Gradient Descent and WarpGrad from the same initialisation across three task loss surfaces, showing representation-space and model-parameter trajectories.
  • Results: WarpGrad learns a geometry that is robust to difficult initialisations and smoothly warps the native loss surface.

E OMNIGLOT

The Omniglot experiment evaluates Warp-Leap on held-out alphabet tasks using convolutional warp-layers inserted between convolutional blocks. Offline replay-buffer mini-batching substantially accelerates meta-training, while the benchmark compares Warp-Leap with gradient-based and other meta-learning baselines.

  • Dataset and setup: Omniglot contains 46 retained alphabet tasks, with 10 alphabets held out for final meta-testing and each task posed as 20-way classification.
  • Model: Warp-Leap inserts convolutional warp-layers between convolutional blocks and finds that simple warp-layers perform well, with added capacity improving generalisation.
  • Evaluation: The evaluation compares Warp-Leap with SGD, KFAC, Leap, Reptile, MAML, and multi-headed fine-tuning on held-out tasks after 100 training steps.
  • Training: Offline training collects 2000 task parameterisations into a replay buffer before repeatedly updating warp parameters from sampled mini-batches.
  • Training: Replay-buffer mini-batching makes meta-training converge much faster without discernible instability.

F ABLATION STUDY: WARP LAYERS, META-OBJECTIVE, AND META-TRAINING

The ablation study varies meta-training objectives, online versus offline protocols, learning-rate meta-learning, and warp-layer architectures to assess WarpGrad design choices.

  • Evaluation protocol: Warp-Leap evaluates these alternatives with 25 pretraining tasks and reports final test accuracy over four independent runs.All ablations share hyper-parameters except online meta-training, which uses a different learning rate.
  • Meta-objective and meta-training: The study compares approximate and full meta-objectives, online and offline meta-training, and whether to meta-learn a scalar task-adaptation learning rate.Warp parameters can learn layer-wise scaling; online meta-training uses a distinct learning rate.
  • Warp-layer architecture: Warp-layer architectures range from channelwise scaling to nonlinear residual designs, two stacked convolutions, and a FiLM-equipped two-layer architecture.The convolutional variants use 128 filters in the first warp convolution and 64 in the second.

G ABLATION STUDY: WARPGRAD AND NATURAL GRADIENT DESCENT

This ablation compares WarpGrad with SGD and KFAC and examines whether its learned geometry resembles the Fisher geometry. WarpGrad improves adaptation beyond block-diagonal preconditioning, but its learned covariance structure differs from the Fisher expectation.

  • Geometry analysis: The ablation measures mean activations and the Schatten-1 norm of covariance minus identity before and after warping, averaged across tasks and adaptation steps.These statistics are computed on held-out test data.
  • WarpGrad versus gradient baselines: WarpGrad’s warped geometries facilitate held-out-task adaptation more than SGD or KFAC, with performance improving further beyond block-diagonal preconditioning.The comparison fixes a random initialization and meta-learns only warp parameters.
  • Geometry analysis: WarpGrad-Leap generally produces zero-centered post-warp activations, while post-warp covariances remain dissimilar from identity and vary across layers.Positive pre-warp activations are attributed to ReLU.
  • Relation to natural gradients: The learned correlation structure differs significantly from the Fisher-matrix expectation, indicating behavior distinct from Natural Gradient Descent.The authors suggest WarpGrad may encode a different geometry using global information beyond the task at hand.

H miniIMAGENET AND tieredIMAGENET

The image-classification experiments evaluate standard few-shot and multi-shot task settings on miniImageNet and tieredImageNet using held-out meta-test tasks. WarpGrad outperforms the compared methods in multi-shot tieredImageNet learning speed and final performance.

  • Datasets: miniImageNet contains 100 classes split into 64 meta-training, 16 meta-validation, and 20 meta-test classes.Each class has 600 images.
  • Datasets: tieredImageNet partitions 608 classes into 34 higher-level categories, using 20 categories for meta-training, 6 for validation, and 8 for testing.The hierarchical split increases separation between meta-training and meta-evaluation.
  • Experimental settings: Few-shot tasks use 5-way 1-shot or 5-shot episodes, with 15 validation images per class and 600 meta-validation or meta-test instances.Meta-training uses corresponding 5-way 5-shot or 15-shot settings.
  • Experimental settings: Multi-shot experiments use 10-way, 640-shot tasks with 50 validation images per class and six convolutional blocks.Convolutional widths are tuned from {64, 64, 160, 160, 256, 256}.
  • Results: WarpGrad outperforms Leap, Reptile, and training from scratch in both convergence rate and final test performance on multi-shot tieredImageNet.The figure reports mean learning curves on held-out meta-test tasks and performance during meta-training.

I MAZE NAVIGATION

The maze-navigation experiments apply WarpGrad to recurrent online meta-learning and reinforcement learning. Warp-RNN converges faster and achieves higher cumulative reward than the baselines while retaining the baseline task-learner size and hidden-state size.

  • Maze-navigation task: Each maze task randomly selects a goal location, which remains fixed during a 200-step episode until the agent reaches it and is teleported.The task therefore requires repeated goal-directed navigation within one episode.
  • Warp-RNN method: Warp-RNN uses a frozen-for-most-training LSTM warp-layer to modulate the task RNN, forming a HyperNetwork-style recurrent preconditioner.The task-adaptable parameters remain those of the baseline RNN.
  • Ablations and comparisons: Updating the meta-LSTM at every task adaptation performs worse than WarpGrad-RNN, while linear T-Nets-RNN provides a comparison to nonlinear preconditioning.These controls assess meta-LSTM update frequency and the form of preconditioning.
  • Warp-RNN method: Warp-RNN meta-training continuously accumulates meta-gradients and updates warp parameters every 30th task-parameter update.The approximate objective is used for the continuous stream of episode batches.
  • Results: Warp-RNN converges faster and achieves higher cumulative reward than all baselines with the same task-learner parameter count and hidden-state size.The implementation adds four lines of code to a standard training script.

J META-LEARNING FOR CONTINUAL LEARNING

This section evaluates WarpGrad as an optimiser for continual learning, where online updates must learn changing sub-tasks while retaining earlier performance without storing past data. The experiment uses continual sine regression and a meta-objective that averages losses across current and preceding sub-tasks.

  • Online SGD often adapts to the most recently encountered data distribution, producing catastrophic forgetting.WarpGrad is investigated as a way to avoid this problem in fully online learning without retaining past data.
  • Continual Sine Regression: Continual sine regression splits [−5, 5] into five consecutive sub-tasks presented sequentially, with 20 gradient steps and batches of five samples per sub-task.The full sequence therefore uses K = 100 online adaptation steps, with each update based only on data from the current sub-task.
  • Model: The task-learner is a four-layer ReLU network augmented with residual warping blocks whose parameters are meta-learned and fixed during adaptation.Each warping block is a two-layer network with 100 hidden units and tanh nonlinearities.
  • Meta-learning an optimiser for continual learning: The continual-learning meta-objective averages validation losses for the current and every preceding sub-task, giving equal weight to tasks that should be learned or remembered.Losses from earlier sub-tasks recur in the objective across later sequence positions, encouraging joint performance over the sequence.
  • Evaluation: The evaluation compares sub-task sequences presented in the meta-training order with sequences presented in random order.Figure 12 reports average log-loss over 100 randomly sampled tasks, while Figure 13 shows predictions before adaptation and after adaptation at successive sub-tasks.
Loading 1909.00025v2…