Source-linked AI summary
Differentiable plasticity: training plastic neural networks with backpropagation
Thomas Miconi, Jeff Clune, Kenneth O. Stanley
TL;DR
The paper asks how agents can continue learning efficiently from experience after initial training, addressing the limits of fixed knowledge and repeated retraining. It trains recurrent networks to optimize both connection weights and Hebbian plasticity with gradient descent, finding strong performance across memorization, Omniglot, and maze exploration tasks. The results position differentiable plasticity as a potential approach to learning to learn.
Problem
Artificial agents often require extensive retraining when tasks change, motivating methods for fast, efficient lifelong learning from ongoing experience.
Method
The paper uses recurrent networks with fixed weights, lifetime Hebbian traces, and trainable plasticity coefficients optimized by gradient descent.
Results
Plastic networks outperform advanced non-plastic recurrent networks on complex pattern memorization, perform competitively on Omniglot, and improve maze exploration.
Takeaways & Limitations
Differentiable plasticity provides a gradient-based route for optimizing meta-properties of neural systems and may support efficient meta-learning with compact added parameters.
Takeaways & Limitations
Understanding the implications of differentiable plasticity requires a broad program of further research across meta-learning and other domains.
Abstract
from arXiv · showhide
How can we build agents that keep learning from experience, quickly and efficiently, after their initial training? Here we take inspiration from the main mechanism of learning in biological brains: synaptic plasticity, carefully tuned by evolution to produce efficient lifelong learning. We show that plasticity, just like connection weights, can be optimized by gradient descent in large (millions of parameters) recurrent networks with Hebbian plastic connections. First, recurrent plastic networks with more than two million parameters can be trained to memorize and reconstruct sets of novel, high-dimensional 1000+ pixels natural images not seen during training. Crucially, traditional non-plastic recurrent networks fail to solve this task. Furthermore, trained plastic networks can also solve generic meta-learning tasks such as the Omniglot task, with competitive results and little parameter overhead. Finally, in reinforcement learning settings, plastic networks outperform a non-plastic equivalent in a maze exploration task. We conclude that differentiable plasticity may provide a powerful novel approach to the learning-to-learn problem.
1. Introduction: the problem of “learning to learn”
The paper addresses how artificial agents can learn quickly from ongoing experience rather than relying on fixed knowledge and repeated retraining. It proposes optimizing synaptic plasticity with backpropagation and evaluates this approach across memorization, one-shot classification, and reinforcement learning.
- Artificial agents typically learn one complex task through extensive training, after which their knowledge remains fixed and changing tasks requires retraining.
- Lifelong learning is important for environments with changing, unpredictable, or initially unknowable features.Autonomous learning could let an agent acquire knowledge of alphabets unavailable to the designer during training.
- Storing fixed task structure in the agent can leave only instance-specific parameters to be learned from a few or even one experience.
- Biological lifelong learning primarily relies on synaptic plasticity, including Hebbian strengthening of connections driven by neural activity.
- The paper extends backpropagation to optimize both baseline connection weights and the amount of plasticity in each connection.
- Across three task types, plastic networks achieve competitive Omniglot results, improve maze exploration, and outperform LSTMs by orders of magnitude on complex pattern memorization.The evaluated tasks are complex pattern memorization, one-shot classification, and reinforcement learning.
2. Differentiable plasticity
Differentiable plasticity combines fixed connection weights with lifetime-changing Hebbian traces, while gradient descent optimizes the structural parameters that govern plasticity. The framework supports multiple Hebbian rules and can represent memories with different durations.
- The framework separates plastic and non-plastic components while allowing multiple Hebbian rules to be implemented.
- Each connection combines a fixed weight with a plastic Hebbian trace scaled by a trainable plasticity coefficient.The effective connection is the baseline weight plus the Hebbian trace multiplied by α_i,j.
- Neuron outputs are computed by applying a nonlinear function to the sum of incoming fixed and plastic contributions.The paper uses tanh, and “inputs” denotes neurons connected to the target neuron.
- Connections can be fully fixed, fully plastic, or mixed depending on the values of the baseline weight and plasticity coefficient.
- Hebbian traces reset at each episode, whereas weights, plasticity coefficients, and the plasticity learning rate are optimized between episodes.
- Weight decay prevents runaway Hebbian feedback but causes traces and memories to decay without input; alternative rules such as Oja’s rule can support stable long-term memories.
3. Related work
Related meta-learning approaches use recurrent state, external memory, fixed or homogeneous plasticity, optimized learning rules, arbitrary update mechanisms, gradient-based adaptation, or learned embeddings. Trainable synaptic plasticity offers a flexible alternative in which connections themselves can store information and plasticity varies by connection.
- Recurrent networks can incorporate past experience within an episode, while external memory architectures read and write ongoing experience through attention.
- Homogeneous-plasticity networks use the same non-trainable plasticity across connections, whereas this framework trains connection-specific plasticity.
- Other approaches optimize learning-rule parameters, compute weight updates dynamically, or use MAML to enable within-episode gradient-based fine-tuning.
- Embedding-based methods reduce classification to comparing test and example-instance embeddings, as in Matching Networks.
- Trainable synaptic plasticity may let connections store information, reducing the computational burden on neurons used for both storage and computation.
- Unlike methods limited to fast stereotyped weight changes, differentiable plasticity may support a broader repertoire, including memories lasting much longer.
4. Experiments and Results
The experiments test differentiable plasticity on pattern memorization, natural-image reconstruction, one-shot Omniglot classification, and maze reinforcement learning. Plastic networks learn novel patterns rapidly, achieve competitive Omniglot accuracy with little parameter overhead, and improve maze performance.
- The experiments evaluate differentiable plasticity on content-addressable memorization, one-shot classification, and reinforcement learning.The memorization experiments include arbitrary high-dimensional patterns and natural images; the classification task uses Omniglot, and the reinforcement-learning task uses maze exploration.
- 4.1. Pattern memorization: Binary patterns: The binary-pattern task presents five 1,000-element patterns repeatedly, then requires reconstructing a randomly selected pattern with half its bits removed.The network receives three randomized presentations of the patterns before reconstructing the complete degraded input.
- 4.1. Pattern memorization: Binary patterns: <1% error is reached within about 200 episodes on 1,000-bit pattern memorization across 10 runs.Error is defined as the proportion of bits with the wrong sign.
- 4.2. The importance of being plastic: a comparison with non-plastic recurrent networks: Mean error falls below .01 within 2,000 episodes for the plastic network, whereas the LSTM needs about 500,000 episodes and the non-plastic RNN flatlines.This comparison uses 50-bit patterns, two patterns per episode, and 2,050-neuron non-plastic baselines; the plastic network uses the same parameters with 51 neurons.
- 4.3. Pattern memorization: Natural images: The trained plastic network memorizes and reconstructs previously unseen natural images, while its learned plasticity matrix contains structured, nonhomogeneous patterns.The reported structure reflects neighboring-pixel correlations and the half-field-zeroing test setup; the authors hypothesize some bands support rapid clearing.
- 4.4. One-shot pattern classification: Omniglot task: 98.3% overall accuracy is achieved on 5-way, 1-shot Omniglot classification, with 320 added plastic parameters over 111,426 total parameters.The median accuracy across 10 runs is 98.5%, and the result is described as competitive with recent methods.
- 4.5. Reinforcement learning: Maze exploration task: Differentiable plasticity strongly improves maze-learning performance, while homogeneous-plasticity networks settle on a low performance plateau.The curves suggest that simple RNNs become stuck in a suboptimal strategy and that connection-specific plasticity is important in this task.
5. Discussion and conclusion
The experiments show that differentiable plasticity can support efficient meta-learning, with strong task-dependent gains and competitive Omniglot performance despite compactness. Fully understanding the approach’s implications remains an open research program.
- Gradient descent can optimize plasticity without requiring an evolutionary process, enabling deep-learning methods to search for plastic structures.
- Differentiable plasticity vastly outperforms alternative meta-learning options on some tasks.
- On Omniglot, differentiable plasticity performs competitively despite its simplicity and compactness in added parameters.
- A comprehensive understanding of the implications of differentiable plasticity requires a broad program of research.
S1. Code example for differentiable plasticity
The code example extends a standard recurrent neural network with trainable fixed weights, plasticity coefficients, Hebbian traces, and gradient-based optimization. Plasticity-specific implementation requires fewer than four additional lines over a simple RNN.
- The example defines and meta-trains a recurrent neural network with plastic connections using PyTorch.
- Plasticity-specific code requires less than four additional lines on top of a simple RNN implementation.
- Fixed weights w and plasticity coefficients alpha are trainable parameters optimized jointly with Adam.
- Each episode initializes neuron activations and Hebbian traces, generates inputs and targets, runs recurrent updates, and applies backpropagation through the episode.
S2. Details for the image reconstruction task
The natural-image reconstruction experiment uses normalized CIFAR10 images in a 1,025-neuron fully connected recurrent network. Episodes present repeated image sequences before testing reconstruction from a degraded stimulus, with parameters updated by gradient descent.
- Stimuli are 32 × 32 CIFAR10 natural images normalized within the [−1, 1] range.
- The network has 1,025 neurons, with one neuron per image pixel plus a bias neuron.
- Each episode presents three images three times in succession, with random order within presentations and 20 time steps per image.
- Blanked pixels receive no direct input, while other neurons update recurrently and plastic connections update during each time step.
- At episode end, PyTorch computes gradients over alpha_i,j, w_i,j, and eta, then Adam updates the parameters with learning rate 1e-4.
S3. Discussion of the plasticity structure in trained image-reconstructing networks
The trained image-reconstructing network develops structured plasticity coefficients rather than homogeneous plasticity. The proposed mechanism both reconstructs missing image regions and removes residual activity from prior stimuli.
- The trained plasticity-coefficient matrix has significant structure, contrasting with homogeneous plasticity in traditional Hopfield-style models.
- The network is proposed to clear remnant activity from previous stimuli while reconstructing the missing image portion.
- Small positive coefficients across opposite image halves let neurons in blank regions use information from the informative half.
- Thin diagonal bands of high positive plasticity reflect correlations between neighboring natural-image pixels.
- Large negative plasticity bands connect pixels to distant regions within the same half, while positive bands connect across halves.
- Hebbian traces store observed pre- and post-synaptic correlations, allowing plastic connections to influence firing from historical activity.
- Negative plasticity is hypothesized to help erase residual information from prior activity during partial-pattern completion.