Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting

Xialei Liu; Marc Masana; Luis Herranz; Joost Van de Weijer; Antonio M. Lopez; Andrew D. Bagdanov

Source-linked AI summary

Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting

Xialei Liu, Marc Masana, Luis Herranz, Joost Van de Weijer, Antonio M. Lopez, Andrew D. Bagdanov

arXiv:1802.02950v4Feb 8, 2018cs.CV

TL;DR

Sequential task learning can cause catastrophic forgetting, while EWC’s practical diagonal-Fisher assumption may discard important parameter correlations. The paper reparameterizes networks through factorized rotations that approximately diagonalize the Fisher Information Matrix, making EWC more effective. Across MNIST, CIFAR-100, CUB-200, and Stanford-40, the rotated method improves over standard EWC and is competitive with exemplar-free state-of-the-art methods.

Problem
Sequential learning can cause catastrophic forgetting, and EWC’s diagonal Fisher assumption may discard important information needed to prevent it.
Method
The method reparameterizes network layers by rotating parameter space through intermediate features so the Fisher Information Matrix becomes more compact and approximately diagonal.
Results
R-EWC consistently improves over EWC, with absolute accuracy gains of 2.1% to 5% on larger two-task datasets while retaining similar second-task accuracy.
Takeaways & Limitations
Rotating parameter space makes EWC more effective at reducing catastrophic forgetting and yields performance comparable to or better than other exemplar-free weight-consolidation methods.
Takeaways & Limitations
The approach still relies on EWC’s diagonal approximation, which is motivated partly by computational constraints and can be unrealistic in the original parameter space.

Abstract

from arXiv · show

In this paper we propose an approach to avoiding catastrophic forgetting in sequential task learning scenarios. Our technique is based on a network reparameterization that approximately diagonalizes the Fisher Information Matrix of the network parameters. This reparameterization takes the form of a factorized rotation of parameter space which, when used in conjunction with Elastic Weight Consolidation (which assumes a diagonal Fisher Information Matrix), leads to significantly better performance on lifelong learning of sequential tasks. Experimental results on the MNIST, CIFAR-100, CUB-200 and Stanford-40 datasets demonstrate that we significantly improve the results of standard elastic weight consolidation, and that we obtain competitive results when compared to other state-of-the-art in lifelong learning without forgetting.

I. INTRODUCTION

Lifelong learning presents tasks sequentially without retaining access to earlier task data, making catastrophic forgetting a central challenge. The paper addresses EWC’s diagonal-Fisher limitation through a rotation-based reparameterization that improves forgetting prevention.

I. INTRODUCTION: Lifelong learning trains networks on sequential task groups while exposing them to data from only the current group.This differs from conventional training, where examples from all expected tasks are generally available.
I. INTRODUCTION: Catastrophic forgetting occurs when adapting network weights to new tasks causes previously learned tasks to be forgotten.The paper frames preventing this interference as the main challenge of sequential learning.
I. INTRODUCTION: Existing lifelong-learning methods either retain exemplars from earlier tasks or avoid storing prior training data through regularization-based approaches such as EWC.Related methods include exemplar rehearsal, inequality constraints, output regularization, and feature preservation.
I. INTRODUCTION: EWC selectively regularizes parameters using the Fisher Information Matrix, but assumes that this matrix is diagonal, a condition the paper identifies as almost never true.The diagonal assumption makes EWC practical but can misrepresent important parameter directions.
I. INTRODUCTION: The proposed method rotates parameter space without changing the forward-pass output, making the backward-pass Fisher Information Matrix approximately diagonal for more effective EWC.The paper reports significantly improved prevention of catastrophic forgetting and performance over standard EWC.

III. ELASTIC WEIGHT CONSOLIDATION

EWC frames sequential task learning as Bayesian updating and approximates the posterior around previously learned parameters with a Gaussian shaped by the Fisher Information Matrix. Its practical diagonal approximation converts a full quadratic regularizer into parameter-wise penalties.

III. ELASTIC WEIGHT CONSOLIDATION: EWC learns task K by combining the new task likelihood with the posterior formed from the preceding K−1 tasks.This sequential Bayesian factorization treats the previous posterior as the prior for the new task.
III. ELASTIC WEIGHT CONSOLIDATION: Because the true posterior is intractable, EWC uses a Laplace approximation that represents it as a Gaussian around previously learned parameters.The Fisher Information Matrix approximates the inverse covariance in this approximation.
III. ELASTIC WEIGHT CONSOLIDATION: In practice, EWC assumes the Fisher Information Matrix is diagonal, replacing the quadratic regularizer with scaling by diagonal entries.This approximation reduces storage and computation from O(N^2) to O(N), where N is the number of parameter-space elements.
III. ELASTIC WEIGHT CONSOLIDATION: The Fisher Information Matrix measures how parameter-space directions relate to forgetting after a network reaches a trained configuration.EWC penalizes movement in high-Fisher-information directions and favors movement along lower-information directions.

B. Limitations of EWC

EWC’s diagonal Fisher approximation is computationally efficient but can be unrealistic in the original parameter space because it discards parameter correlations. Rotating the space improves diagonal alignment and preserves more Fisher-matrix energy.

B. Limitations of EWC: The diagonal Fisher approximation reduces computation and storage from O(N^2) to O(N), but may be unrealistic for sequential learning in the original parameter space.The paper identifies efficiency as the practical reason for using the diagonal assumption.
B. Limitations of EWC: Rotating parameter space to align average log-probability gradients with coordinate axes makes the diagonal approximation more reasonable.In the rotated space, EWC can optimize the new task while better preserving the old task.
B. Limitations of EWC: 40.8% of the Fisher-matrix energy is retained by the diagonal in a second-layer MNIST multilayer perceptron example.The full matrix is clearly non-diagonal, so the diagonal approximation misses significant correlations between weights.

IV. ROTATED ELASTIC WEIGHT CONSOLIDATION

The method reparameterizes the network so its forward response is unchanged while the Fisher Information Matrix becomes more nearly diagonal, making EWC more effective.

IV. ROTATED ELASTIC WEIGHT CONSOLIDATION: The desired reparameterization preserves the network’s feedforward response while better satisfying EWC’s diagonal-FIM assumption.The resulting diagonal FIM can be efficiently estimated in the new parameter space.
IV. ROTATED ELASTIC WEIGHT CONSOLIDATION: The approach performs gradient-descent minimization for new tasks in the reparameterized space.EWC is therefore applied using the approximately diagonal FIM of that space.
IV. ROTATED ELASTIC WEIGHT CONSOLIDATION: SVD-based direct rotation is impractical because it is expensive, ignores sequential network structure, and requires an unavailable FIM.These issues motivate an indirect rotation implemented within the network architecture.

A. Indirect rotation

Indirect rotation uses local feature transformations to reparameterize fully connected layers without changing forward computation, while concentrating more FIM energy on diagonal terms.

A. Indirect rotation: Assuming independence between output gradients and inputs factorizes the FIM into one factor for backpropagated gradients and another for inputs.This factorization motivates separate rotations of input and output spaces.
A. Indirect rotation: The rotations are implemented as two additional fixed local linear layers, x′ = U1x and y = U2y′.Because the rotations are local, they can be integrated into the network architecture.
A. Indirect rotation: After reparameterization, training estimates the FIM and learns W′ using transformed inputs and outputs.The procedure is otherwise the same as the original EWC training procedure.
A. Indirect rotation: The method preserves both networks’ forward passes by expressing W as U2W′U1, keeping the sequential structure intact.The new learning problem uses transformed inputs, outputs, and weights.
A. Indirect rotation: The layer-wise extension applies the same procedure using each layer’s inputs and backpropagated output gradients.This assumes a block-diagonal FIM and produces layer-specific reparameterizations.

B. Extension to convolutional layers

The fully connected rotation procedure extends to convolutional layers by applying channel-space rotations through additional 1 × 1 convolutional layers and processing kernel slices.

B. Extension to convolutional layers: Convolutional rotations use two additional 1 × 1 convolutional layers to transform the input and output channel spaces.This adapts the fully connected construction with only slight modifications.
B. Extension to convolutional layers: For each spatial coordinate, input and output fibers are d1-dimensional and d2-dimensional vectors, respectively.Their self-correlation matrices are averaged over spatial coordinates.
B. Extension to convolutional layers: A mode-i fiber is the vector obtained by fixing every tensor index except i.A tensor slice fixes all indices except two.
B. Extension to convolutional layers: Algorithm 1 incrementally fits each task with EWC using rotated parameters and a Fisher matrix, then combines parameters before computing new rotations.The task data are supplied as per-class training sets.
B. Extension to convolutional layers: The convolutional procedure computes decompositions, rotates kernel slices, and tiles the resulting slices into the final rotated kernel tensor.The rotated slices are defined from the original kernel’s spatial slices.

V. EXPERIMENTAL RESULTS

The paper evaluates the proposed approach against EWC and other baselines across multiple experiments.

V. EXPERIMENTAL RESULTS: The experiments compare the proposed approach with EWC and other baselines.The passage introduces the experimental evaluation without reporting outcomes.

A. Experimental settings

The evaluation uses sequential class-incremental tasks across MNIST, CIFAR-100, CUB-200 Birds, and Stanford-40 Actions, with dataset-specific preprocessing and architectures. Experiments assume task labels are unknown at inference.

Datasets: Each dataset is divided equally into class groups that are learned as sequential tasks.The evaluated datasets are MNIST, CIFAR-100, CUB-200 Birds, and Stanford-40 Actions.
Datasets: CUB-200 images use cropped bounding boxes resized to 224×224, while Stanford-40 images are resized to 256×256 and randomly cropped to 224×224.
Training details: For the fine-grained datasets, the method finetunes an ImageNet-pretrained VGG-16 with global pooling added after the final convolutional layer.The modification reduces memory use and computational complexity.
Inference setting: The experiments use a single network head because task labels are unknown at inference, increasing output neurons as new tasks are added.

B. Disjoint MNIST comparison and ablation study

On disjoint MNIST, R-EWC outperforms fine-tuning and EWC across regularization settings, while rotating fully connected layers is sufficient for substantial improvement. Larger regularization preserves the first task but makes the second harder to learn.

MNIST comparison: R-EWC clearly outperforms FT and EWC for every tested λ value.The best trade-off can vary depending on which layers are rotated.
Trade-off parameter: Lower λ values behave more like fine-tuning, causing task 1 to be forgotten more quickly.
Ablation study: All R-EWC configurations that rotate fully connected layers significantly outperform EWC.
Conclusion: R-EWC maintains similar task 2 performance to EWC while producing much less catastrophic forgetting on task 1.The regularized FIM is usually between 10^-2 and 10^-4, with λ=100 suggested as a more balanced trade-off.

C. Comparison with EWC on two tasks

Across larger datasets split into two sequential tasks, R-EWC improves overall accuracy over EWC while forgetting less on the first task and retaining similar second-task accuracy. The multi-task comparison extends this advantage as additional tasks are learned.

Two-task comparison: R-EWC gains 2.1% to 5% absolute accuracy over EWC across all evaluated larger datasets.
Two-task comparison: R-EWC forgets significantly less on the first task while attaining similar accuracy on the second task.
More-task comparison: With four tasks, accuracy decreases for both methods as more tasks are learned, but R-EWC consistently outperforms EWC.
More-task comparison: On Stanford-40 Actions, the performance margin between R-EWC and EWC grows as more tasks are learned.
More-task comparison: For each previous Stanford-40 task after training the fourth task, R-EWC avoids forgetting better than EWC.

E. Comparison with the state-of-the-art

On CIFAR-100, R-EWC outperforms the compared exemplar-free baselines and approaches, but does not reach, an upper bound requiring all previous data. The paper attributes its broader advantage to reparameterizing the FIM so EWC can work more effectively.

State-of-the-art comparison: R-EWC outperforms FT, EWC, LwF, and EG on the four-task CIFAR-100 comparison.
State-of-the-art comparison: R-EWC gains about 5% over EWC and achieves better performance than LwF.
State-of-the-art comparison: R-EWC remains below the UpperBound, which trains newer tasks with all previous-task images available.The UpperBound cannot be updated for new tasks under the described setting.
Conclusion: The method reparameterizes layers to produce a more compact and more diagonal FIM, addressing information discarded by EWC’s diagonal approximation.
Conclusion: The fixed additional layers approximate parameter-space rotation without increasing the number of parameters.
Conclusion: Across several tasks and settings, R-EWC consistently improves over EWC and obtains results comparable to or better than other exemplar-free weight-consolidation methods.