Continual Learning Through Synaptic Intelligence

• ANNs are typically trained on a dataset and then frozen, so adapting to changed data distributions requires retraining and risks catastrophic forgetting.Source: §1 · p. 1
• Biological neural networks continually acquire knowledge from non-stationary data, suggesting that biological mechanisms absent from ANNs may support continual learning.Source: §1 · p. 1
• Biological synapses use molecular machinery across spatial and temporal scales, but few studies had shown how this complexity benefits learning in ANNs.Source: §1 · p. 1
• The proposed synapses track past and current parameters while estimating each synapse’s importance for solving earlier tasks.Source: §1 · p. 1
• Penalizing changes to important synapses lets future tasks rely mainly on previously unimportant synapses, reducing forgetting during sequential classification learning.Source: §1 · p. 1

Architectural approaches

• Architectural methods reduce task interference by freezing or modifying weights, but network-copying approaches grow in complexity with the number of tasks.Approaches include freezing weights, reducing shared-layer learning rates, changing nonlinearities, sparsifying gradients, and augmenting copied networks.Source: §2 · p. 2

Functional approaches

• Functional methods preserve old input-output mappings through distillation or hidden-activation regularization, requiring an additional old-network forward pass for each new data point.Source: §2 · p. 2

Structural approaches

• Structural methods penalize parameter changes using measures such as EWC’s Fisher-weighted quadratic penalty, but exact diagonal Fisher computation scales linearly with output dimensionality.Source: §2 · p. 2

3. Synaptic framework

• Online synaptic importance protects parameters that contributed strongly to prior loss reduction, allowing less important parameters to adapt to new tasks.The framework seeks a local, online structural regularizer whose importance measure is maintained at individual synapses or parameters.Source: §3 · pp. 2–4

Importance estimation

• Parameter-specific importance is computed from gradient-update contributions accumulated along the entire training trajectory, where the path integral equals the total loss change.The gradient field is decomposed by parameter, giving each parameter an interpretable contribution to changes in total loss.Source: §3 · p. 3

Noise and calibration

• SGD noise typically overestimates per-parameter importance, so the strength parameter is generally chosen below one to compensate.Source: §3 · pp. 3–4

Surrogate loss

• The quadratic surrogate matches the original task’s loss decrease, net parameter displacement, and endpoint minimum rather than using a Hessian approximation at the minimum.These three conditions uniquely determine the surrogate quadratic loss summarizing the descent trajectory.Source: Figure 2 · p. 3, §3 · p. 3

Regularization

• The regularized objective pulls important parameters toward their previous-task reference values, with a strength parameter c trading off retention of old memories against learning new tasks.A damping parameter ξ bounds the regularization expression when parameter displacement approaches zero.Source: §3 · pp. 3–4

Implementation

• Importance estimates are updated continuously during each task, while cumulative importance and reference weights update only at task boundaries before per-task importance is reset.Source: §3 · p. 4

Scope

• The surrogate-loss motivation is exact for two tasks, although the method is evaluated empirically for learning additional tasks.Source: §3 · p. 4

Relation to EWC

• Compared with EWC, the framework estimates importance online over the full learning trajectory instead of using a separate end-of-task Fisher-information estimate.Source: §3 · p. 4

Quadratic training analysis

• The time-integrated matrix Q is independent of the descent time constant τ and can be expressed in the Hessian eigenbasis.The derivation uses eigenvalues, eigenvectors, and initial-to-final parameter discrepancies to obtain an explicit formula for Q.Source: §4 · p. 4

Averaged initial conditions

• After averaging over random initial conditions, Q reduces to the Hessian up to a scale factor removed by the normalization.The scale factor reflects the discrepancy between initial and final conditions, and the normalization removes it at zero damping.Source: §4 · pp. 4–5

Special cases

• For a diagonal Hessian, normalized Q becomes the diagonal Hessian, while for a rank-1 Hessian it becomes the diagonal of a non-diagonal low-rank Hessian.In both special cases, normalization removes the scale of movement in parameter space.Source: §4 · p. 5

Continual-learning implication

• Low-rank structure leaves many synaptic-weight directions unconstrained by a task, leaving excess capacity for future-task modification without interfering with old-task performance.This identifies the continual-learning relevance of the low-rank Hessian case.Source: §4 · p. 5

Scope and limitation

• For general losses with a Hessian that varies along the trajectory, no simple correspondence between importance and the endpoint Hessian is expected, although the measures are empirically correlated.The authors relate this correlation to the comparable effectiveness of endpoint-based measures in later experiments.Source: §4 · p. 5

5.1. Split MNIST

• Consolidation preserved high accuracy on earlier split-MNIST tasks, whereas networks without it fell to chance after learning all digits.On Task 2, both models performed near 1, but only consolidation kept average accuracy across Tasks 1 and 2 close to 1; after all tasks, unprotected early-task accuracy dropped to chance.Source: §5.1 · p. 6

5.2. Permuted MNIST benchmark

• Synaptic consolidation retained high permuted-MNIST accuracy across nine additional tasks and performed only slightly worse than simultaneous training, with results comparable to EWC.The figure compares consolidation and EWC against SGD variants as task count increases.Source: Figure 4 · p. 6, §5.2 · p. 7
• Without consolidation, second-layer importance correlations increased across permuted-MNIST tasks; consolidation kept these synapse sets largely uncorrelated.The authors associate correlated importances with interference and catastrophic forgetting.Source: §5.2 · pp. 6–7

5.3. Split CIFAR-10/CIFAR-100 benchmark

• On split CIFAR-10/100, consolidation produced similar validation accuracy across tasks and outperformed non-consolidated training except on the last task.The comparison used a CNN trained first on CIFAR-10 and then on five CIFAR-100 tasks, with five repeated runs.Source: §5.3 · p. 7
• Consolidated networks matched or exceeded single-task-from-scratch validation accuracy on most tasks, while non-consolidated networks showed the opposite training-versus-validation pattern.The exception was Task 1, CIFAR-10, which had 10× more examples per class.Source: §5.3 · p. 7

6. Discussion

• The method alleviates catastrophic forgetting by estimating each synapse’s importance for past tasks and penalizing changes to important synapses.This allows novel tasks to be learned with minimal interference to previously learned tasks.Source: §6 · p. 7
• Unlike EWC’s offline Fisher estimate at a task’s loss minimum, this approach computes consolidation strength online across the entire parameter trajectory.Despite this distinction, the two methods yielded similar permuted-MNIST performance.Source: §6 · p. 7
• The model requires synapses to function as higher-dimensional dynamical systems that accumulate task-relevant information and retain previous parameter values.The paper contrasts this with artificial synapses represented by single scalar weights.Source: §6 · p. 8
• The authors present this higher-dimensional synaptic model as one specific solution that motivates broader research into biologically inspired control of learning dynamics.They explicitly make no claim that biological synapses behave like the model.Source: §6 · p. 8

A. Split CIFAR-10/100 CNN architecture

• The CIFAR-10/100 experiments used the default CIFAR-10 CNN from Keras.Source: A. Split CIFAR-10/100 CNN architecture · p. 9
• Table 1 summarizes the split CIFAR-10/100 model architecture and hyperparameters.Source: Table 1 · p. 9
• In the table, m denotes the number of splits.Source: Table 1 · p. 9

B. Additional split CIFAR–10 experiments

• The CNN was trained for 60 epochs on each half of split CIFAR-10 using Adam, with identical control and consolidation experiments repeated 10 times.Task A comprised the first five categories and Task B the remaining five categories.Source: B. Additional split CIFAR–10 experiments · pp. 9–10
• Consolidation significantly improves validation performance on both Task A and Task B compared with no consolidation.The comparison uses repeated experiments with error bars representing uncertainty across 10 runs.Source: Figure 7 · p. 10, B. Additional split CIFAR–10 experiments · p. 10
• Approximately 4.5% higher validation accuracy on Task B suggests consolidation also improves transfer learning.The effect was observed consistently for consolidation values in the range 0.1 < c < 10.Source: B. Additional split CIFAR–10 experiments · p. 10

other metrics

• Prior sensitivity measures primarily use local objective curvature, with the Hessian carrying computational overhead and potentially being negative definite.The paper motivates alternatives that are more efficient for online use.Source: other metrics · p. 10
• The empirical Fisher approximates parameter sensitivity using gradients at the current parameters and avoids the model-distribution labels required by the full Fisher.Computing the full Fisher online would require at least one additional backpropagation pass.Source: other metrics · p. 10
• The path-integral approach evaluates parameter importance over a trajectory rather than at a single final parameter value.This distinction makes it proportional to the Hessian diagonal for a quadratic, whereas the empirical Fisher is zero at the minimum.Source: other metrics · p. 10