On Learning the Geodesic Path for Incremental Learning

1. Introduction

• Incremental learning presents tasks gradually, and sequential learning can degrade performance on previously learned concepts, creating a stability-plasticity dilemma.Limited memory for preserving knowledge further constrains the model as it adapts to new tasks.Source: §1 · p. 1
• Prior methods use memory replay and distillation, but output-space distillation [25] and feature-space cosine distillation do not account for gradual change between tasks.The paper motivates GeoDL with the idea of gradual change in human learning.Source: §1 · p. 2
• GeoDL proposes distilling features along a geodesic flow between task-specific subspaces, introducing an information-geometry-based loss for incremental learning.The method projects feature sets from two different networks and distills along the path between their subspaces.Source: Figure 1 · p. 1, §1 · p. 2
• GeoDL’s geodesic-path distillation yields less forgetful networks than distillation losses using the Euclidean metric.Source: §1 · p. 2

Representative memories

• Representative-memory methods replay selected past exemplars, with herding and prototype-based selection among the approaches.Liu et al. [26] use meta-learning to update memory, generative replay [39] requires additional networks, and Iscen et al. [20] adapt features to retain exemplars efficiently.Source: §2 · p. 2

Gradient trajectories

• Gradient-trajectory methods constrain updates to reduce forgetting: EWC [22] uses Fisher information, GEM [27] constrains gradient similarity, and Riemannian walk [4] generalizes EWC.These approaches seek trajectories with low error on both previous and new tasks or allow positive backward transfer.Source: §2 · p. 2

Parameter updates

• Parameter-update methods restrict which network parameters change, using dropout or attention, task-specific masking or pruning, and meta-learning to balance tasks.Mallya et al. [28] [29] use masking or pruning for different tasks; Rajasegaran et al. [36] propose meta-learning to maintain equilibrium across tasks.Source: §2 · p. 2

Regularization with distillation losses

• Prior distillation preserves old-task outputs or features, but does not model their underlying manifolds; this work uses cosine-distance classifier weights, unlike the fully connected setting in.Output distillation [25], balanced fine-tuning with temporary classifier-layer distillation [2], and feature-space Euclidean losses are prior strategies; Belouadah and Popescu argue distillation can hurt when at least a few examples are available.Source: §2 · p. 2

Geodesic flows

• This method uses geodesic flow for knowledge distillation, preserving learned-feature similarity across intermediate subspaces along the geodesic path.Geodesic flow via projection to intermediate subspaces models gradual task changes; such projections have also been used in domain adaptation and few-shot learning.Source: §2 · p. 2

3.1. Class incremental learning setting

• Each task provides data for sequential learning, while a fixed memory budget retains only a limited number of examples for later task updates.Training begins with D0; thereafter, the data combines the current task with memory, and later tasks often contain fewer samples and novel categories.Source: §3.1 · p. 3
• At each task, the model updates from its previous parameters using current-task data plus memory, while its classifier expands as novel classes arrive.The network maps inputs to descriptors through an encoder and then to predictions through a classifier.Source: §3.1 · p. 3
• Incremental-learning evaluation uses average accuracy to assess learning new tasks and forgetting rate to measure catastrophic forgetting; success requires high accuracy and low forgetting.Forgetting compares initial-task test accuracy under the initial model Φ0 and final model ΦT.Source: §3.1 · p. 3

3.2. Regularization with knowledge distillation

• LwF [25] distills predictions by minimizing differences between old and new model outputs, preserving predictions on previously seen classes during adaptation.Distillation can be combined with cross-entropy [26] or triplet loss [48].Source: §3.2 · p. 3
• Hou et al. use a cosine-embedding less-forget constraint on latent features, but these prediction- and feature-based losses do not exploit each task’s manifold structure.The cosine distance directly applies the Euclidean metric between features from the two models.Source: §3.2 · p. 3

4. Proposed Method

• GeoDL addresses prior distillation’s failure to model gradual task changes and feature-space geometry by enforcing consistency along the geodesic connecting the previous and current models.Source: §4 · p. 3

4.1. Gradual walk with intermediate subspaces

• Features are embedded in low-dimensional subspaces on the Grassmann manifold, and the geodesic between old and current subspaces lets projections capture smooth changes between tasks.PCA can provide the subspace basis.Source: §4.1 · p. 4

4.2. Projection to the intermediate subspaces

• GeoDL distills cosine similarity between old and current features projected along the geodesic; its manifold-structure matrix defines the loss, with conventional distillation recovered when the subspaces match.When P t−1 = P t, Q = I, yielding the conventional distillation loss.Source: §4.2 · pp. 4–5

4.3. Classifiers and exemplars selection

• Classes use normalized template vectors trained with cross-entropy, while memory exemplars are selected by herding samples nearest the mean-embedding classifier; end-to-end training learns smoothly transitioning embeddings.Source: §4.3 · p. 6

5.1. Dataset and implementation

• Experiments use CIFAR-100, ImageNet-subset, and ImageNet-full, with random-seed class orders and three runs.The initial sequence uses 50 classes for CIFAR-100 and ImageNet-subset, or 500 for ImageNet-full; subsequent evaluations use 5, 10, or 25 tasks.Source: §5.1 · p. 6, §5 · p. 6

5.2. Evaluations

• Comparisons include LwF [25], iCARL, BiC [47], LUCIR, Mnemonics [26], and a joint-CNN upper bound.GeoDL is also combined with LUCIR by replacing its feature-distillation loss with GeoDL’s loss while retaining cross-entropy and margin-ranking losses.Source: §5.2 · p. 6
• On CIFAR-100, GeoDL improves iCARL by 8%, 13%, and 15% and LUCIR by 2%, 5%, and 5% for 5, 10, and 25 tasks, respectively.The paper also reports lower forgetting rates than recent incremental-learning algorithms across task counts.Source: §5.2 · p. 6
• Across the evaluated datasets, GeoDL achieves high average accuracy and prevents catastrophic forgetting, while outperforming prior incremental-learning methods on ImageNet-full.Figure 4 plots accuracy by task for CIFAR-100, ImageNet-subset, and ImageNet-full; Table 1 reports average accuracy and forgetting rate on ImageNet-subset.Source: §5.2 · pp. 6–7, Figure 4 · p. 7, Table 1 · p. 7

5.3. Ablation studies

• On CIFAR-100 and ImageNet-subset, GeoDL performs best with AME, and all distillation losses achieve better accuracy with AME than with CNN or k-NME.GeoDL improves incremental learning without additional losses; its gains are smaller with CNN because past-task classifiers are not updated during new-task learning.Source: Figure 5 · p. 8, §5.3 · p. 8
• Across exemplar counts, GeoDL outperforms LwF [25] with feature distillation; more exemplars help close the gap between cross-entropy alone and added distillation.With more exemplars, CNN achieves higher performance than k-NME or AME, which the paper attributes to balanced per-class exemplar counts.Source: Table 2 · p. 8, §5.3 · p. 8
• GeoDL is robust to the subspace dimension n, and the authors generally set n = 127.Table 3 reports average accuracy for 10-task CIFAR-100 experiments across subspace dimensions.Source: Table 3 · p. 8, §5.3 · p. 8

• GeoDL uses the geodesic path between current- and old-task representations to account for gradual changes between consecutive tasks and smooth the learning transition.The paper presents GeoDL as a novel distillation loss for incremental learning.Source: §6 · p. 8
• GeoDL achieves competitive results against the state of the art across various incremental-learning datasets and consistently improves existing baselines.Source: §6 · p. 8
• Ablation studies show that GeoDL performs better than previous distillation losses for incremental learning.Source: §6 · p. 8

• The old-model subspace P_{t−1}, current-model subspace P_t, and orthogonal complement R are used to compute the geodesic flow at ν.Source: §7 · p. 11
• A generalized SVD is used to obtain the orthonormal matrices U_1 and U_2.Source: §7 · p. 11
• Intermediate geodesic steps ν ∈ (0, 1) project features Π(ν)⊤z for the distillation-loss similarity, but a closed-form solution avoids computing or storing every intermediate projection.Source: §7 · p. 11
• The values λ_1, λ_2, and λ_3 are calculated from diagonal elements of Γ(1), with ω_i = arccos(γ_i) and γ_i clamped to [−1, 1] for computational stability.Source: §7 · p. 11

ImageNet-subset comparison

• On ImageNet-subset, GeoDL improves the basic IL approach using only cross-entropy and herding exemplars, compared with LwF [25] and LUCIR across task counts and classifiers.Source: §8 · p. 11

Impact of increasing the number of exemplars in the memory

• GeoDL outperforms other distillation methods across memory sizes; with 20 exemplars, LGeoDL beats LLwF by 1.6% and LCos by 7.4%.AME obtains the highest accuracy under most memory sizes on ImageNet-subset, and more exemplars help close the gap between training with and without distillation.Source: Table 4 · p. 11, §8 · p. 11

Analyzing the similarity between models

• GeoDL yields the highest CCA similarity to the base model on CIFAR-100 and ImageNet-subset, indicating stronger retention of base-class representations [35].Scores use feature outputs from the last layer and samples from the base classes; higher CCA scores indicate less forgetting.Source: §8 · p. 12, Figure 6 · p. 12, Figure 7 · p. 12

Time and memory consumption

• Subspace SVD costs O(n^2d) and geodesic flow O(nd); iterations are 1.4× slower than LwF [25] and 1.3× slower than LUCIR, using 2.4GB versus 2.1GB (LCos) and 1.7GB (LLwF).GeoDL does not require additional memory to store exemplars.Source: §8 · p. 12

Initialization with less number of classes

• With 50-class initialization, GeoDL scores 62.8% versus LUCIR's 60.5%; with 10 classes, scores are 60.5% versus 57.3%.Source: §8 · p. 12