Metaplasticity as adaptive gradient preconditioning for incremental learning

Isabelle Aguilar; Zayn Andre Zainal; Omid Kavehei

Source-linked AI summary

Metaplasticity as adaptive gradient preconditioning for incremental learning

Isabelle Aguilar, Zayn Andre Zainal, Omid Kavehei

arXiv:2608.14634v1Jul 27, 2026cs.LG

TL;DR

Continual learning systems struggle to retain prior knowledge while adapting to non-stationary streams. SynGAP reframes metaplasticity as task-free online geometric gradient preconditioning and substantially outperforms regularization and replay baselines.

Problem
Artificial neural networks face a fundamental stability-plasticity dilemma when learning continuously from non-stationary streams, while regularization methods can trade retention for adaptation as tasks increase.
Method
SynGAP maintains an online exponential moving average of diagonal Fisher information and maps it to a bounded multiplicative mask that attenuates gradients along historically important directions.
Results
SynGAP prevents catastrophic forgetting on non-stationary streams and substantially outperforms standard adaptive optimizers, traditional regularization, and replay-based baselines.
Takeaways & Limitations
SynGAP enables continuous consolidation without explicit task boundaries, offline Fisher recomputation, or complex regularization-scaling heuristics.
Takeaways & Limitations
SynGAP uses a diagonal Fisher approximation that ignores off-diagonal dependencies and still relies on a sampling buffer for the active-learning signal.

Abstract

from arXiv · show

Biological intelligence naturally prevents catastrophic forgetting through Complementary Learning Systems (CLS) theory, a macroscopic consolidation process driven at the local level by synaptic metaplasticity: the continuous, history-dependent neuromodulation of individual synapses. While artificial neural networks struggle with the stability-plasticity dilemma in non-stationary environments, existing solutions often require task labels or incur massive memory overhead, diverging from biological reality. Re-framing this localized neuromodulation as an optimization-driven process, we introduce $\textbf{SynGAP}$: $\textbf{Syn}$aptic $\textbf{G}$eometric $\textbf{A}$daptive $\textbf{P}$reconditioning. SynGAP is a task-free continual learning framework based on adaptive gradient preconditioning. Rather than relying on explicit episodic triggers, SynGAP simulates real-time metaplasticity by maintaining an exponential moving average of the Fisher Information Matrix over a continuous data stream. During the optimization step, these dynamic metaplastic states are translated into a bounded multiplicative mask that preconditions raw gradients, selectively attenuating updates to critical historical parameters. Empirical evaluations demonstrate SynGAP's superior ability to mitigate catastrophic forgetting compared to established baselines. On the Split CIFAR-100 benchmark, SynGAP delivers a $4\times$ increase in accuracy compared to EWC++ and outperforms Experience Replay (ER) by almost $10\%$, while reducing the forgetting measure by over $10\%$ against both methods. Furthermore, on the CORe50 benchmark, SynGAP achieves about $68\%$, a $10\%$ improvement over optimizer baselines. By mathematically formalizing continuous biological metaplasticity as stable gradient-based regularization, SynGAP offers a highly robust and memory-efficient solution for adaptive intelligence at the edge.

Introduction

SynGAP reframes biological synaptic metaplasticity as bounded, geometry-aware gradient preconditioning for task-free continual learning. It continuously protects historically important parameters while preserving plasticity, avoiding task boundaries and extensive replay storage.

Introduction: Catastrophic forgetting arises because standard gradient descent optimizes immediate performance, overwriting representations needed for historical tasks.CLS theory explains biological mitigation through hippocampal buffering followed by gradual neocortical integration.
Introduction: Existing continual-learning approaches face capacity limits, task-boundary requirements, memory overhead, distribution-shift stability gaps, or rigid regularization.These limitations span architectural, replay-based, and regularization strategies.
Introduction: SynGAP formalizes metaplasticity as geometry-aware optimization on a partially observable Riemannian manifold, using curvature to stabilize continual learning.Preconditioning can also accelerate convergence by adjusting for varying loss curvatures.
Introduction: SynGAP is a task-free framework that simulates continuous neuromodulation by online tracking and directly preconditioning raw stochastic gradients.Its mask attenuates updates along directions of high historical curvature while preserving plasticity along flat, non-interfering trajectories.
Introduction: A bounded multiplicative mask balances stability and plasticity by preventing parameter freezing while avoiding the severe memory footprint of episodic replay.SynGAP continuously tracks parameter importance without task boundaries or extensive data storage overhead.

Related Works

Related work addresses catastrophic forgetting through parameter regularization, gradient-based optimization interventions, and biologically inspired metaplasticity. These approaches motivate SynGAP’s dynamic, local protection of historical knowledge during updates rather than static global loss penalties.

Parameter regularization: Parameter-regularization methods restrict weight plasticity according to estimated importance for previous tasks and add penalties that pull weights toward consolidated values.Their main distinction is how parameter importance is estimated, including EWC.
Limitations of regularization: Regularization methods can create scaling issues as task counts grow because they treat consolidation as a static loss constraint rather than a dynamic learning property.This produces a stability-plasticity trade-off between retaining prior knowledge and adapting to new distributions.
SynGAP motivation: SynGAP applies local metaplastic updates governed by g(F), modulating optimization steps according to Fisher Information of the joint loss.The framework is presented as an alternative to globally updating on the newest task data, which can struggle to find a joint minimum.
Optimization-based methods: Gradient preconditioning methods intervene directly in optimization by scaling updates using historical gradients or loss geometry, while gradient projection seeks directions orthogonal to old-task gradients.These approaches aim to reduce interference without relying solely on global loss penalties.
Biological inspiration: Biological metaplasticity provides a history-dependent model in which synaptic internal states regulate future plasticity, embedding prior consolidation into synaptic update dynamics.This neuroscience perspective motivates moving beyond transient optimizers and computationally expensive curvature matrices.

Preliminaries

Continual learning trains across sequential, task-specific datasets with limited access to prior data, creating a stability-plasticity challenge. The preliminaries define the Fisher Information Matrix and gradient preconditioning as tools for protecting historical knowledge.

Continual learning setting: Continual learning presents tasks sequentially, with training on the current dataset while previous-task data is typically restricted to a small memory buffer.The objective is to learn the current distribution without severely degrading performance on earlier distributions.
Continual learning setting: The overall objective is to minimize expected risk across all tasks observed so far, approximating joint optimization despite lacking full access to past datasets.Continual-learning methods must approximate the joint loss over all datasets.
The Fisher information matrix: The Fisher Information Matrix measures parameter importance for past data and is approximated diagonally for computational feasibility.The theoretical FIM is defined as the covariance of the score function; the formulation uses a diagonal FIM, F ∈ R^|θ|.
Gradient preconditioning: Gradient preconditioning transforms the standard gradient-descent update, while this framework uses it to attenuate updates in directions critical for preserving historical knowledge.This reverses the stated use of adaptive optimizers such as Adam, which accelerate optimization in high-curvature directions.

Methodology

SynGAP estimates parameter importance continuously with an online diagonal Fisher EMA, then converts this state into a bounded mask that preconditions gradients during optimization. Its parallel tracking and learning streams decouple metaplastic gating from raw gradient computation.

Online Fisher information tracking: SynGAP continuously updates parameter importance using an exponential moving average of the diagonal Fisher Information Matrix over the current data stream.The retention rate α controls metaplastic-state persistence, while the instantaneous Fisher information is evaluated on each current batch.
Bounded metaplastic preconditioning: The plasticity function g(F) transforms unbounded Fisher-EMA states into a consolidation mask Ω bounded within [ε, 1 + ε].The function uses Tikhonov damping for numerical stability and ambient plasticity ε to prevent complete intransigence.
Optimization step: During optimization, SynGAP simultaneously computes raw task gradients and updates the historical Fisher-EMA state through parallel streams.The resulting mask gates the raw gradients before they modify the network.
Optimization step: The final update is θ_t+1 = θ_t − η(g(F) ⊙ ∇_θL(θ)), applying element-wise metaplastic preconditioning without an additive loss penalty.Here, η is the global learning rate and ⊙ denotes the element-wise Hadamard product.

Experiments

SynGAP is evaluated on task-free class-incremental and continuous domain-incremental benchmarks using accuracy, forgetting, and intransigence metrics. It outperforms established baselines, while ablations show that bounded preconditioning and slower metaplastic retention are important for preserving historical knowledge.

Benchmarks: SynGAP is tested on Split CIFAR-100’s 10-task class-incremental stream and CORe50’s continuous New Classes scenario without task identifiers or boundary signals.Split CIFAR-100 contains 100 classes split into 10 tasks; CORe50 comprises 50 classes in a highly non-stationary environment.
Evaluation Metrics: Evaluation reports Average Accuracy (ACC), Average Forgetting (FM), and Intransigence (INT) to assess performance, catastrophic forgetting, and resistance to acquiring new information.ACC measures mean stream performance after training, FM measures accuracy loss from peak to final performance, and INT compares performance with a jointly trained upper bound.
Split CIFAR-100: 27.28% ACC and 37.87% FM: SynGAP outperforms ER’s 17.60% ACC on Split CIFAR-100 while reducing forgetting relative to severe baseline degradation.SGD, SGDM, Adam, Ballistic, and EWC++ achieve final accuracies below 9% and forgetting measures exceeding 51%.
CORe50: 68.65% accuracy and 30.85% FM: SynGAP leads CORe50, surpassing ER’s 67.22% accuracy and improving forgetting relative to ER’s 33.72% and SGD’s 41.56%.EWC++ reaches 40.51% accuracy and 60.53% forgetting on CORe50.
Intransigence Trade-off: SynGAP maintains INT scores of 6.91% on CIFAR-100 and 4.09% on CORe50, balancing historical retention against the plasticity–stability trade-off.SGDM and Adam achieve negative Split CIFAR-100 INT values of −5.90% and −5.20%, whereas EWC++ reaches 5.67% INT on CORe50.
Ablations: 14.72% ACC and 49.46% FM: replacing the bounded plasticity mask with an unbounded linear penalty degrades performance from the Full Model’s 27.28% ACC and 37.87% FM.Removing preconditioning produces 17.60% accuracy and 54.23% FM, while lowering α produces 19.27% accuracy and 46.32% FM, showing the importance of bounded scaling and slower retention decay.

Discussion

The discussion frames SynGAP as multiplicative, task-free gradient preconditioning that preserves the current-stream optimization landscape, complements experience replay, and enables continuous consolidation. It also identifies diagonal-Fisher assumptions and richer geometric approximations as directions for future improvement.

Optimization landscape: SynGAP’s bounded tanh-based multiplicative preconditioning avoids the global optimization-landscape distortion associated with additive EWC-style penalties.Traditional regularization can produce gradients that scale disproportionately with the number of tasks.
Experience replay: SynGAP complements Experience Replay by gating new-data updates away from critical dimensions of previously learned tasks.This addresses gradient interference between memory-buffer updates and high-magnitude novel-stream gradients.
Continuous consolidation: Task-free online Fisher estimation enables continuous consolidation with minimal computational overhead, without discrete task boundaries or offline Fisher re-computations.This operational design targets continuous real-world data streams.
Limitations: SynGAP’s diagonal Fisher approximation ignores off-diagonal dependencies, motivating block-diagonal or Kronecker-factored approximations for complex loss landscapes.The diagonal approximation is described as computationally efficient and sufficient, but potentially incomplete geometrically.
Future directions: Future work will investigate hierarchical timescales and extend the scalar plasticity mask Ω with second-order off-diagonal constraints.Such extensions could remove the ambient plasticity term ε and enable parameter lockdown without risking dead networks.

Conclusion

SynGAP is a task-free continual-learning framework that re-conceptualizes biological metaplasticity as online geometric gradient preconditioning. It protects historical knowledge by continuously adapting gradient updates without explicit task boundaries or additive loss penalties.

Framework: SynGAP re-conceptualizes biological metaplasticity as online geometric gradient preconditioning for task-free continual learning.The framework is designed to operate without explicit task boundaries.
Mechanism: SynGAP maintains a continuous exponential moving average of the Fisher Information Matrix to represent evolving synaptic information.This provides the online state used for gradient adaptation.
Mechanism: A globally normalized bounding function projects the Fisher-based state into selective gradient attenuation that protects historical knowledge.The attenuation is applied during optimization rather than through an additive loss penalty.
Design properties: SynGAP avoids both explicit task boundaries and additive loss penalties while implementing its continual-learning protection mechanism.These design choices support task-free operation as described in the conclusion.