Source-linked AI summary
On Learning the Geodesic Path for Incremental Learning
Christian Simon, Piotr Koniusz, Mehrtash Harandi
TL;DR
Incremental learning must acquire new tasks without forgetting previously learned concepts, yet sequential updates can degrade past performance. The paper introduces GeoDL, which distills features along geodesic paths between low-dimensional task manifolds rather than comparing responses directly. GeoDL is reported to preserve past knowledge more effectively, outperform prior distillation techniques, and improve existing baselines, with computational costs tied to subspace and geodesic-flow construction.
Problem
Sequentially updating neural networks for new tasks can degrade performance on previously learned concepts, making catastrophic forgetting a central incremental-learning challenge.
Method
GeoDL constructs low-dimensional representations for current and previous tasks and distills features along the geodesic path connecting their subspaces.
Results
GeoDL consistently improves existing baselines and outperforms prior knowledge-distillation techniques across various datasets.
Takeaways & Limitations
Geodesic-path distillation provides a smooth transition between task representations for preserving past knowledge during incremental learning.
Takeaways & Limitations
Computing GeoDL requires O(n^2d) time for subspace extraction and O(nd) time for geodesic flow.
Abstract
from arXiv · showhide
Neural networks notoriously suffer from the problem of catastrophic forgetting, the phenomenon of forgetting the past knowledge when acquiring new knowledge. Overcoming catastrophic forgetting is of significant importance to emulate the process of "incremental learning", where the model is capable of learning from sequential experience in an efficient and robust way. State-of-the-art techniques for incremental learning make use of knowledge distillation towards preventing catastrophic forgetting. Therein, one updates the network while ensuring that the network's responses to previously seen concepts remain stable throughout updates. This in practice is done by minimizing the dissimilarity between current and previous responses of the network one way or another. Our work contributes a novel method to the arsenal of distillation techniques. In contrast to the previous state of the art, we propose to firstly construct low-dimensional manifolds for previous and current responses and minimize the dissimilarity between the responses along the geodesic connecting the manifolds. This induces a more formidable knowledge distillation with smooth properties which preserves the past knowledge more efficiently as observed by our comprehensive empirical study.
1. Introduction
Incremental learning requires models to acquire new tasks while preserving prior knowledge, but sequential updates can cause catastrophic forgetting. The paper proposes GeoDL, which uses geodesic paths between task representations for distillation.
- Incremental learning presents objectives and data gradually, unlike conventional learning where they are available from the beginning.
- Sequentially learning new tasks can degrade performance on previously learned concepts, creating a stability-plasticity dilemma.
- Existing approaches use memory replay and feature or output distillation to reduce catastrophic forgetting, but distillation-loss design remains open.
- GeoDL proposes distilling features along a geodesic path between low-dimensional subspaces representing different task models.
- GeoDL is reported to yield less forgetful networks than distillation losses based on the Euclidean metric.
2. Related Work
Related work addresses catastrophic forgetting through representative-memory replay and regularization of parameter-update directions. These approaches preserve earlier knowledge under limited access to past data.
- Representative memories: Representative-memory methods select exemplars from past categories because incremental learning cannot access the entire historical dataset.
- Representative memories: Herding selects potential exemplars, while feature-adaptation strategies seek to retain exemplars efficiently in memory.
- Gradient trajectories: Gradient-trajectory methods regularize update directions, including EWC, which uses the Fisher Information Matrix to balance previous and new tasks.
3. Preliminaries
The paper formulates class-incremental learning as sequential model updates using current-task data plus a limited memory, evaluated by accuracy and forgetting. Knowledge distillation preserves prior predictions or features, while prior feature losses ignore task-manifold structure.
- Class incremental learning setting: Class-incremental learning stores only a limited number of examples in memory and trains later tasks using current data together with that memory.
- Class incremental learning setting: The model expands its classifier as new classes arrive while updating parameters from the previous model using current-task data and memory.
- Evaluating an IL Model: Average accuracy measures learning new tasks, whereas forgetting rate measures degradation relative to the initial model's performance.
- Regularization with knowledge distillation: Distillation minimizes changes to shared parameters or old-model predictions so previously learned knowledge remains stable during adaptation.
- Regularization with knowledge distillation: Prior cosine-based feature distillation compares model features directly with a Euclidean metric and does not exploit each task's manifold structure.
4. Proposed Method
GeoDL models old and current feature spaces as low-dimensional subspaces and enforces distillation along their geodesic path. The method combines this geometry-aware loss with exemplar-based incremental-learning training and classification.
- 4.1. Gradual walk with intermediate subspaces: GeoDL models feature sets from the old and current models as low-dimensional subspaces on the Grassmann manifold.A basis for each subspace can be obtained using PCA.
- 4.1. Gradual walk with intermediate subspaces: The method defines a geodesic flow between the old and current subspaces to capture smooth changes between tasks.Features are projected along intermediate subspaces on this geodesic path rather than compared only with a Euclidean metric.
- 4.2. Projection to the intermediate subspaces: GeoDL distills knowledge by comparing old and current feature encodings after projection along the geodesic flow.The resulting inner product uses a positive semi-definite matrix Q that defines the manifold structure between features from different tasks.
- 4.2. Projection to the intermediate subspaces: The proposed distillation loss is a geodesic-flow generalization of cosine similarity that accounts for smooth transitions between task manifolds.The conventional distillation loss is recovered when the old and current subspaces coincide, giving Q = I.
- Classifier and exemplar selection: The end-to-end algorithm uses a classifier with normalized class templates and selects memory exemplars nearest to the mean embedding by cosine distance.An adaptive weighting scheme is used for GeoDL during training.
- Training procedure: Training combines cross-entropy with GeoDL, samples from current data and stored exemplars, and updates exemplars using herding.The algorithm computes PCA subspaces, generates the geodesic flow, projects features, minimizes both losses, and evaluates incrementally.
5. Experiments
The experiments compare GeoDL with state-of-the-art incremental-learning methods across CIFAR-100 and ImageNet variants, evaluating accuracy and forgetting. Ablations examine classifiers, exemplar-memory size, and geodesic subspace dimension.
- 5. Experiments: The evaluation compares GeoDL with LwF, iCARL, BiC, LUCIR, and Mnemonics on CIFAR-100, ImageNet-subset, and ImageNet-full.A joint-CNN upper bound is also reported, and GeoDL is combined with LUCIR by replacing its feature-distillation loss.
- 5.2. Evaluations: Figure 4 tracks accuracy as learned classes increase for five- and ten-task settings across CIFAR-100, ImageNet-subset, and ImageNet-full.The x-axis is the number of classes learned at a given time, and the y-axis is the corresponding accuracy.
- 5.2. Evaluations: GeoDL achieves high average accuracy and prevents catastrophic forgetting in the reported evaluations.The experiments assess incremental learning using average accuracy and forgetting rate.
- 5.3. Ablation studies: In ablations, GeoDL performs best with the AME classifier and improves incremental-learning performance without additional losses.The comparison varies CNN, k-NME, and AME classifiers while using cross-entropy and distillation losses.
6. Conclusions
The paper presents GeoDL, a distillation loss for incremental learning that uses geodesic paths between old and current representations. It reports competitive results, consistent baseline improvements, and better performance than prior distillation losses.
- 6. Conclusions: GeoDL uses the geodesic path between current-task and old-task representations to create a smooth transition during incremental learning.The method is designed to prevent catastrophic forgetting as tasks change gradually.
- 6. Conclusions: GeoDL achieves competitive incremental-learning results across several datasets and consistently improves existing baselines.The conclusion also reports that GeoDL outperforms prior knowledge-distillation techniques.
- 6. Conclusions: Ablation studies show that GeoDL performs better than previous distillation losses for incremental learning.
7. The details of generating the geodesic flow
The geodesic flow is generated from the old and current model subspaces and their orthogonal complement. Intermediate points along this flow provide projections used to compute the distillation similarity without storing every intermediate projection.
- 7. The details of generating the geodesic flow: The method computes geodesic flow from the old-model subspace, current-model subspace, and their orthogonal complement.These subspaces are used to compute the flow at parameter ν.
- 7. The details of generating the geodesic flow: Generalized SVD of the relevant subspace products produces the orthonormal matrices U1 and U2 used in the geodesic construction.
- 7. The details of generating the geodesic flow: All intermediate geodesic time steps ν ∈ (0, 1) project features with Π(ν)^Tz to obtain the similarity used by the distillation loss.A closed-form solution avoids computing or storing every intermediate projection.
- 7. The details of generating the geodesic flow: The method derives λ1, λ2, and λ3 from Γ(1), computes ωi = arccos(γi), and clamps γi to [-1, 1] for stability.
- 7. The details of generating the geodesic flow: Algorithm 2 summarizes the procedure for generating the geodesic flow.
8. Additional results
Additional experiments evaluate GeoDL across memory sizes, representation similarity, and computational costs. The method outperforms competing distillation losses across memory sizes and achieves the highest CCA similarity, while incurring extra computation.
- Memory size: GeoDL outperforms other knowledge-distillation techniques under all tested memory sizes on ImageNet-subset.With 20 exemplars, LGeoDL exceeds LLwF by 1.6% and LCos by 7.4%.
- Memory size: Using more exemplars narrows the performance gap between training without distillation and training with a distillation loss.
- Representation similarity: GeoDL produces the highest CCA similarity between current and base-model feature representations on both CIFAR-100 and ImageNet-subset.CCA is computed from last-layer feature outputs and evaluates preservation of base-model representations.
- Time and memory consumption: Obtaining a subspace with standard SVD costs O(n^2d), while obtaining the geodesic flow costs O(nd).
- Time and memory consumption: Each GeoDL iteration is 1.4× slower than LwF and 1.3× slower than LUCIR, while requiring 2.4GB of computational memory.LCos and LLwF require 2.1GB and 1.7GB, respectively.