PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta Learning Backpropagation And Improving It

Louis Kirsch, Jürgen Schmidhuber

arXiv:2012.14905v4cs.LGcs.AIcs.NEstat.ML

TL;DR

Meta-learning seeks to learn learning algorithms, but many approaches retain fixed, human-designed inner-loop components. The paper proposes VSML, using sparse weight sharing and tiny LSTMs to learn algorithms in recurrent dynamics. It reports backpropagation implementation and new algorithms that generalize to unseen datasets and learn through fast association.

  • Problem

    Many meta-learned algorithms focus on similar train and test tasks or still depend on fixed, human-designed inner-loop components such as backpropagation.

    Source: §1 · p. 1

  • Method

    VSML uses sparsity and weight sharing, replacing neural-network weights with tiny LSTMs to learn learning algorithms in recurrent dynamics.

    Source: §1 · p. 1, §8 · p. 10

  • Results

    VSML meta-learns supervised algorithms that do not explicitly compute gradients and generalize to unseen datasets; introspection finds fast association qualitatively different from stochastic gradient descent.

    Source: §8 · p. 10

  • Takeaways & Limitations

    VSML RNNs can implement backpropagation and its parameter updates implicitly in recurrent dynamics, predicting on MNIST without a human-designed gradient-computation graph.

    Source: §8 · p. 10

    The work is a proof of concept on small-scale problems, with performance not yet matching mini-batched large-data settings; its current variant also costs O(WN^2).

    Source: §7 · p. 10

Abstract

from arXiv · show

Many concepts have been proposed for meta learning with neural networks (NNs), e.g., NNs that learn to reprogram fast weights, Hebbian plasticity, learned learning rules, and meta recurrent NNs. Our Variable Shared Meta Learning (VSML) unifies the above and demonstrates that simple weight-sharing and sparsity in an NN is sufficient to express powerful learning algorithms (LAs) in a reusable fashion. A simple implementation of VSML where the weights of a neural network are replaced by tiny LSTMs allows for implementing the backpropagation LA solely by running in forward-mode. It can even meta learn new LAs that differ from online backpropagation and generalize to datasets outside of the meta training distribution without explicit gradient calculation. Introspection reveals that our meta learned LAs learn through fast association in a way that is qualitatively different from gradient descent.

1 Introduction

Meta-learning aims to learn learning algorithms, but many approaches still depend on fixed, human-designed inner-loop components. VSML uses sparsity and weight sharing to meta-learn algorithms, including backpropagation in recurrent dynamics, and reports generalization to datasets outside its training distribution.

1 Introduction

  • Many meta-learning approaches target similar train and test tasks, while broad generalization across datasets or environments remains important for replacing human-engineered learning algorithms.Some recent approaches generate more general algorithms, but many still rely on human-designed, unmodifiable inner-loop components such as backpropagation.Source: §1 · p. 1
  • VSML replaces neural-network weights with tiny LSTMs and uses sparsity and weight sharing to express learning algorithms in recurrent dynamics.This lets it implement backpropagation in forward-mode and meta-optimize backpropagation-like algorithms.Source: §1 · p. 1
  • VSML meta-learns online learning algorithms from scratch that often learn faster than gradient descent and generalize beyond the meta-training distribution, such as from MNIST to Fashion MNIST.The paper describes it as the first neural meta-learner without hard-coded backpropagation to show such strong generalization.Source: §1 · p. 2

2 Background

The background distinguishes approaches that modify network weights to implement learning rules from Meta RNNs, which encode a learning algorithm in recurrent activations. Each offers a different balance of update flexibility, parameterization, and available learning memory.

2 Background

  • Meta-learning without fixed inner-loop gradient descent has used either learnable weight-update mechanisms or black-box recurrent models.The paper groups these approaches as fast weight programmers or learned learning rules (FWPs / LLRs) and Meta RNNs.Source: §2 · p. 2
  • FWPs / LLRs modify network weights through an explicit update architecture, whose meta-variables can be reused across many learned variables.The framework includes fast weight programmers, hypernetworks, synaptic plasticity, and learned learning rules.Source: §2 · p. 2
  • Meta RNNs encode the learning algorithm in their trained weights and use activations as learning memory, but their parameter count can greatly exceed that memory.The resulting overparameterization can make them prone to overfitting and task-specific solutions rather than general learning algorithms.Source: §2 · p. 2

3 Variable Shared Meta Learning (VSML)

VSML reuses shared parameters in sparse, connected sub-RNNs whose distinct states can represent neural-network weights and whose messages support neural computation. The paper describes meta-learning algorithms from scratch and training VSML to implement backpropagation through its recurrent dynamics.

3 Variable Shared Meta Learning (VSML)

  • Each sub-RNN corresponds to a neural-network weight, with shared parameters, distinct states, and message passing between sub-RNNs.The arrangement can also be viewed as a sparse shared weight matrix or as complex neurons; the input and output dimensions correspond to the grid axes.Source: §3 · pp. 2–4, Figure 1 · p. 3
  • Stacking VSML layers ties their parameters while keeping their states untied, and backward messages prevent information from flowing only forward.The architecture can be extended to other neural architectures, and multiple recurrent ticks may be applied per layer.Source: §3 · pp. 4–5

3.1 Meta learning general-purpose learning algorithms from scratch

  • Meta-training optimizes shared LSTM parameters to minimize prediction losses across sequential examples, while meta-testing uses no explicit gradient descent.The algorithm can be meta-trained end-to-end with gradient descent or with gradient-free methods such as evolutionary strategies.Source: §3.1 · p. 5

3.2 Learning to implement backpropagation in RNNs

  • Learning algorithm cloning trains VSML to store weights and biases in its states, compute neural outputs, and update those parameters according to backpropagation.The VSML RNN is optimized to regress outputs, parameter changes, and errors toward targets; repeated forward and reverse layer passes correspond to evaluating and updating the network.Source: §3.2 · p. 6

4 Experiments

The experiments test whether VSML can reproduce backpropagation through recurrent dynamics, learn an online learning algorithm from scratch, and generalize across input configurations and datasets. They also examine how its learned algorithm differs from gradient descent.

4 Experiments

  • The experiments evaluate recurrent implementation of backpropagation, learning from scratch, robustness, and transfer, using a VSML model with about 2,400 meta variables and 257,200 learned variables.The implementation uses LSTMs and the message interpretation from Equation 7.Source: §4 · p. 6

4.1 VSML RNNs can implement backpropagation

  • Cloned backpropagation learning works on MNIST and Fashion MNIST, reaching 87% deep and 90% shallow MNIST test accuracy, and 76% deep and 80% shallow Fashion MNIST test accuracy.Regular gradient descent still outperforms the cloned algorithm; the reported focus is successful cloning rather than performance.Source: §4.1 · pp. 6–7, Figure 5 · p. 6

4.2 Meta learning supervised learning from scratch

  • Meta-trained from scratch on MNIST, VSML learns faster than online gradient descent and generalizes to Fashion MNIST without hard-coded gradients during meta testing.A Meta RNN heavily overfits in comparison.Source: §4.2 · p. 7, Figure 6 · p. 7

4.3 Robustness to varying inputs and outputs

  • All six unseen input and output configurations perform comparably to the unchanged reference, and invariance to random projections suggests learning beyond transfer of learned representations.The configurations were not seen during training.Source: Figure 7 · p. 7, §4.3 · p. 8

4.4 Varying datasets

  • VSML generalizes better than Meta RNNs, Hebbian fast weights, and external memory, including to unseen Random and Sum Sign tasks with no shared structure with the other datasets.In many cases its performance is similar to SGD, but a little more sample efficient early in training.Source: §4.4 · p. 8

5 Analysis

The analysis contrasts VSML’s rapid input-label associations with gradient descent’s slower adaptation. In the illustrated MNIST sequence, VSML usually corrects a failed prediction on the next repeated example, while SGD does not adapt correctly within 20 steps.

5 Analysis

  • VSML’s rapid input-label associations differ qualitatively from SGD, which makes many incorrect predictions and fails to adapt correctly in 20 steps.The analysis reports that almost all failed VSML predictions are followed by correct predictions with high certainty; those associations generalize to future unseen inputs.Source: §5 · p. 9, Figure 9 · p. 9

6 Related Work

The related work distinguishes memory-based meta learners, explicit weight-update methods, gradient-based learned optimizers, program search, and shared-parameter multi-agent approaches. VSML is positioned as using variable sharing to learn general-purpose learning algorithms without fixed inner-loop gradients or a symbolic programming language.

Memory based meta learning (Meta RNNs)

  • Meta RNNs encode a learning algorithm in recurrent weights and use activations as memory, while VSML uses weight sharing to address their overparameterization and overfitting.The passage identifies Meta RNNs as [10] [56].Source: §6 · p. 9

Learned gradient-based optimizers

  • Unlike learned optimizers that take a loss and parameter gradients as inputs, VSML targets learning without fixed gradient calculation in the inner loop.The cited learned-optimizer work is [2] [22] [24].Source: §6 · p. 9

Discrete program search

  • Unlike discrete program search, VSML discovers and encodes general-purpose learning algorithms in variable-shared RNNs without defining a symbolic programming language.The passage notes that neural-network parameter search is usually easier than symbolic program search because of loss-landscape smoothness.Source: §6 · p. 10

Multi-agent systems

  • VSML differs from shared-parameter multi-agent approaches by demonstrating variable sharing for more general-purpose learning algorithms and presenting a weight-update interpretation.The cited multi-agent work includes [32] [18] and [36].Source: §6 · p. 10

7 Discussion and Limitations

The discussion presents VSML as a proof of concept for small-scale problems whose performance does not yet match large-data mini-batched learning. It also identifies computational cost and meta-optimization challenges, while noting that variable sharing may provide regularization and reuse.

7 Discussion and Limitations

  • The current work is a proof of concept limited to small-scale problems, and its performance does not yet match mini-batched learning with large quantities of data.Scaling to harder problems and larger meta-task distributions remains future work.Source: §7 · p. 10
  • The current VSML variant costs more computationally than standard backpropagation, with cost O(WN^2) when a sub-RNN of state size N is used for each of W weights.The authors note small N may make the cost acceptable, and future work may relax the one-sub-RNN-per-weight choice.Source: §7 · p. 10
  • Meta optimization is prone to local minima, and longer input-to-feedback delays make credit assignment harder; backprop-based meta optimization also limits learning horizons.Early experiments suggest diverse meta-task distributions may mitigate these issues, while evolutionary strategies enabled longer horizons and more stable optimization.Source: §7 · p. 10
  • VSML can be viewed as regularizing learning-algorithm weights through a representational bottleneck, although other regularization techniques might also achieve generalization.Unlike most regularizers, VSML also introduces reuse of the same learning principle and permutation invariance through variable sharing.Source: §7 · p. 10

8 Conclusion

VSML uses shared, sparse parameters to meta-learn learning algorithms, including algorithms that learn without explicit gradient computation. Its recurrent dynamics can implement backpropagation and can also learn distinct algorithms whose fast associations support sample-efficient generalization.

8 Conclusion

  • VSML replaces neural-network weights with parameter-sharing tiny LSTMs, a simple weight-sharing and sparsity principle for meta-learning learning algorithms.The recurrent dynamics can implicitly encode backpropagation updates and can also be meta-learned from scratch without explicit gradient computation.Source: §8 · p. 10
  • VSML can meta-learn supervised learning algorithms from scratch that do not explicitly compute gradients and generalize to unseen datasets.Introspection finds fast association qualitatively different from stochastic gradient descent, with gains in sample efficiency.Source: §8 · p. 10

A Derivations

The derivation shows that VSML’s weight matrices can be represented by a single shared, sparse RNN weight matrix. Learning-algorithm cloning then uses a curriculum to train VSML RNNs to reproduce backpropagation in shallow and deep networks.

A Derivations

  • Any weight matrices W and C can be expressed by a single weight matrix ˜W with zero and shared entries.The theorem specifies entries using Cij, Wij, their sum, or zero, according to index conditions.Source: A Derivations · p. 14

B.1 Learning algorithm cloning

  • VSML RNNs can produce tanh-activated multiplicative input-weight interactions with relatively small error, although they do not match the target dynamics everywhere.The learned LSTMs are repeated to obtain an emergent neural network.Source: §B.1 · p. 15, Figure 10 · p. 15
  • A curriculum matches forward activations, backward errors, and weight updates, then trains a deep network first with ground-truth intermediate errors and next with the RNN’s predictions.This addresses accumulated cloning errors and shifts in input and state distributions in deeper networks; the resulting VSML RNN trains shallow and deep configurations.Source: §B.1 · p. 15

B.2 Meta learning from scratch

Across the reported meta-test experiments, VSML learns faster than SGD with Adam at the beginning of training and associates inputs with labels quickly. The section also reports generalization comparisons, an online CIFAR-10 experiment, and a convolutional VSML variant.

Meta testing learning curves & sample efficiency

  • Learning at the beginning is accelerated compared to SGD with Adam across the reported meta-test learning curves.Figure 12 covers six unseen meta-test tasks; Figure 13 reports cumulative accuracy on the first 100 examples.Source: §B.2 · p. 16, Figure 12 · p. 16, Figure 13 · p. 16

Ablation: Projection augmentations

  • Random projection augmentation slightly reduces VSML’s generalization when omitted, while enabling it for baselines does not significantly increase their generalization.VSML benefits from the augmentations; the reported comparison also includes Meta RNN, Hebbian fast weights, and external memory baselines.Source: §B.2 · pp. 16, 18, Figure 16 · p. 18

Introspect longer meta test training run

  • On a 100-step full-MNIST run, VSML still associates inputs with their labels quickly compared to SGD, though learning is slower with greater input variety.The introspection plots show output probabilities after each input and its output error.Source: §B.2 · p. 18, Figure 17 · p. 18

Omniglot

  • VSML generalizes well to unseen datasets such as Fashion MNIST, but learns more slowly on Omniglot while learning significantly faster there than SGD with Adam.The Omniglot evaluation uses unseen alphabets in a 5-way, 1-shot setting.Source: §B.2 · p. 18, Figure 18 · p. 18

Convolutional Neural Networks

  • A convolutional VSML using kernel size 3, stride 2, and 8 channels performs similarly to the fully connected architecture on the existing datasets.The implementation replaces each kernel weight with a multidimensional RNN state and kernel multiplications with VSML sub-RNNs.Source: §B.2 · p. 19, Figure 20 · p. 19

CIFAR10

  • On CIFAR-10, meta-test learning is initially faster than SGD and still generalizes to Fashion MNIST, but learning speed later decreases.The authors attribute the slowdown probably to short-horizon bias and state that further improvements are needed to scale VSML to harder problems; online CIFAR-10 learning requires many consecutive examples.Source: §B.2 · pp. 19–20, Figure 21 · p. 19

C Other Training Details

The implementation uses LSTMs with forward and backward messages, and averages messages to improve meta-training stability. Additional training details specify state handling, batching, clipping, and weight scaling for learning-algorithm cloning.

C Other Training Details

  • VSML RNNs use A · B LSTMs with forward and backward messages; message projections have output sizes 8, with state size 64 for cloning and 16 for meta-learning from scratch.Averaging messages instead of summing them improves meta-training stability.Source: C Other Training Details · p. 20

C.1 Learning algorithm cloning

  • During forward evaluation, LSTM state is frozen; during the backward pass, only two state dimensions for weight and bias are retained, and other message inputs are zeroed except those encoding input and error.A buffer of VSML states is sampled during learning-algorithm cloning, with each new state appended to maintain state diversity.Source: §C.1 · p. 20
  • For cloning, batches of 64 data points are run through the RNN and their weight- and bias-related states are averaged, analogously to batching in gradient descent.LSTM states are also clipped between −4 and 4 during meta testing to prevent exploding states.Source: §C.1 · p. 20
  • Because LSTM hidden states lie between −1 and 1, cloning scales weights and biases down by 4 to store them in the context state.This scaling is only relevant during learning-algorithm cloning.Source: §C.1 · p. 20

C.2 Meta learning from scratch

The experiments produce online learning algorithms with high sample efficiency, though their final performance trails batched SGD. The discussion identifies alternative VSML designs and connections to distributed memory, modular learning, and self-organization.

C.2 Meta learning from scratch

  • Meta-training used evolution strategies across 128 GPUs for 10k steps, while the searched hyperparameters were not further tuned.Source: §C.2 · p. 20
  • The discovered online learning algorithms have high sample efficiency, but their final performance trails batched SGD.Source: §C.2 · p. 21
  • Minimizing errors across training encourages early good predictions; optimizing only final error remains an alternative for prioritizing final performance.Source: §C.2 · p. 21
  • Replacing shared weights with LSTMs can be recursively repeated to create a hierarchy of arbitrary depth, which the paper leaves unexplored.Source: §C.2 · p. 21
  • The study uses message-passing RNNs for sparse shared weights; other sharing and sparsity patterns remain open directions.Source: §C.2 · p. 21

D.1 VSML as distributed memory

  • VSML’s shared meta variables implement memory reading and writing mechanisms distributed across the network.Source: §D.1 · p. 22

D.2 Connection to modular learning

  • Unlike modular-learning experts with unique parameters and conditional selection, VSML modules share meta variables to resemble a learning algorithm.Source: §D.2 · p. 22

D.3 Connection to self-organization and complex systems

  • VSML is framed as a self-organizing system in which many sub-RNNs induce a global learning algorithm.Source: §D.3 · p. 22

Related papers

Loading 2012.14905v4…