Source-linked AI summary
Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning
Ali Cheraghian, Shafin Rahman, Pengfei Fang, Soumava Kumar Roy, Lars Petersson, Mehrtash Harandi
TL;DR
FSCIL requires learning disjoint new classes from only a few examples while predicting across all accumulated classes, making conventional distillation difficult. The paper combines semantic word vectors with knowledge distillation and attention-based visual-semantic alignment, and reports state-of-the-art performance on three datasets. Its evaluation assumes 5-shot novel tasks and notes slightly different first-task results because semantic word vectors are used.
Problem
FSCIL must learn disjoint novel classes from few examples while retaining prior knowledge and producing unified predictions across all tasks.
Method
The method uses word-vector semantics, fixed superclass embeddings, and attention over multiple embeddings to align visual representations and support knowledge distillation.
Results
The method beats existing state-of-the-art approaches on MiniImageNet, CUB200, and CIFAR100, reaching 39.04% last-session accuracy on MiniImageNet versus TOPIC's 24.42%.
Takeaways & Limitations
Class semantics can provide useful auxiliary information that enables knowledge distillation for FSCIL and supports better generalization to novel classes.
Takeaways & Limitations
The evaluation uses 5-shot novel tasks, and semantic word vectors produce slightly different first-task results from the comparison setting.
Abstract
from arXiv · showhide
Few-shot class incremental learning (FSCIL) portrays the problem of learning new concepts gradually, where only a few examples per concept are available to the learner. Due to the limited number of examples for training, the techniques developed for standard incremental learning cannot be applied verbatim to FSCIL. In this work, we introduce a distillation algorithm to address the problem of FSCIL and propose to make use of semantic information during training. To this end, we make use of word embeddings as semantic information which is cheap to obtain and which facilitate the distillation process. Furthermore, we propose a method based on an attention mechanism on multiple parallel embeddings of visual data to align visual and semantic vectors, which reduces issues related to catastrophic forgetting. Via experiments on MiniImageNet, CUB200, and CIFAR100 dataset, we establish new state-of-the-art results by outperforming existing approaches.
1. Introduction
Few-shot class-incremental learning requires adapting to new classes from limited data while retaining performance on previously learned classes. The paper addresses this challenge with semantic word vectors, knowledge distillation, and attention-based visual-semantic alignment.
- Motivation: FSCIL models must learn new classes incrementally from few examples without degrading performance on previously learned classes.The setting is framed as realistic because class-specific data may become available over time, but catastrophic forgetting makes adaptation difficult.
- Motivation: Standard knowledge distillation is difficult in FSCIL because novel classes provide too few samples to learn newly added trainable weights.Previous scores can be preserved, but the limited novel-class data cannot adequately train corresponding parameters.
- Approach: The proposed method uses semantic word vectors to facilitate knowledge distillation for few-shot novel classes.Semantic information supplies auxiliary class-level representations that are cheap to obtain.
- Approach: An attention module merges multiple superclass embeddings to align visual representations with semantic vectors for novel classes.Superclass embeddings are learned from base-task classes and held fixed for subsequent novel classes.
- Contributions: The paper reports new state-of-the-art results on MiniImageNet, CUB200, and CIFAR100.The contributions include semantic-guided distillation, visual-semantic alignment with automatically assigned superclass annotations, and extensive experiments.
2. Related work
Related work distinguishes class-incremental learning from other incremental settings and reviews few-shot incremental learning and knowledge distillation. The paper situates its approach within these areas while presenting an architecture that maps visual features into a semantic domain.
- Incremental learning: Incremental learning is categorized into task-incremental, domain-incremental, and class-incremental settings.Class-incremental learning is the paper's focus because testing requires a unified classifier without task labels.
- Incremental learning: Class-incremental learning predicts among all classes from merged task outputs, whereas task-incremental learning provides task labels at test time.This distinction determines whether the model must infer the correct class across the full accumulated label space.
- Few-shot incremental learning: Few-shot incremental learning extends incremental learning to sequences where later novel tasks contain only a few examples per class.Dynamic few-shot learning is described as similar but limited to a base task followed by one novel task.
- Knowledge distillation: Knowledge distillation is used in incremental learning to preserve old-task knowledge while learning new tasks.Prior approaches apply distillation losses to transfer or retain knowledge across neural networks and incremental classifiers.
- Proposed architecture: The proposed architecture forwards an image through backbone B and mapping module M to estimate a semantic vector from its visual feature.This visual-to-semantic mapping provides the basis for aligning image representations with class semantics.
3. Method
The method combines semantic word vectors, knowledge distillation, and attention over multiple superclass-specialized embeddings to support few-shot class-incremental learning. A frozen backbone, visual-semantic mapping, memory prototypes, and incremental module updates address limited data, forgetting, and overfitting.
- 3.1. Problem Formulation: The task provides many base-task samples but only 5-shots per class for later disjoint tasks, while requiring a unified classifier over current and previous classes.Training at each session uses only the current task’s samples, with semantic class embeddings available during training.
- 3.2. Knowledge Distillation: Knowledge distillation preserves prior classifier scores while new classes are added, using semantic word vectors to address insufficient data for learning novel-class weights.The backbone is trained on the base task and kept frozen later; class prototypes from memory represent previous classes.
- Training procedure: Training uses classification, distillation, and attention losses, while incremental updates fine-tune the mapping and embedding components without adding new parameters.The procedure initializes the backbone and embeddings on the base task, assigns superclass labels to later tasks, and updates memory with class prototypes.
- 3.4. Model Overview: The architecture maps image features into the semantic domain: backbone B produces g, embedding modules produce e, concatenation forms f, and mapping module M produces aligned representation y.The final representation combines global and embedding features before visual-semantic alignment.
- 3.3. Multiple embeddings for few shot tasks: Multiple embeddings are trained as experts for superclass groups derived from base-task word-vector semantics, then assigned to novel classes using those superclass clusters.Attention produces a weighted average of embedding outputs, with weights constrained to sum to 1.
- 3.3. Multiple embeddings for few shot tasks: The attention module is trained to make the fused embedding resemble the module associated with each sample’s superclass, helping limit overfitting on few-shot novel data.Combining specialized embedding features with the global feature supports classification of both base and novel classes.
4. Experiments
The experiments evaluate the method across FSCIL and DFSL benchmarks, ablations, and sensitivity analyses. Across MiniImageNet, CIFAR100, and CUB200, the method outperforms existing approaches, while its components and hyperparameters show distinct effects.
- Experimental setup: The method is evaluated on FSCIL using MiniImageNet, CUB200, and CIFAR100, with additional experiments in the DFSL setting.The study includes ablations and analyses of semantic features, backbones, losses, embeddings, temperature, and supercluster count.
- FSCIL results: 39.04% accuracy is achieved in the last MiniImageNet session, compared with 24.42% for TOPIC.On CIFAR100, the method reaches 34.80% versus 29.37% for TOPIC; on CUB200, it achieves 32.96% in the last session.
- Ablation study: Ld is more effective than La for last-session accuracy: Ld helps retain previous tasks, while La enriches novel-task representations.The novel tasks contain only a few training samples.
- Ablation study: The method exceeds the baseline without multiple embeddings, supporting the value of the embedding module.The study also evaluates alternative semantic vectors and backbones using base-task accuracy, novel-task accuracy, and their harmonic mean.
- Ablation study: GloVe with ResNet101 forgets less of the base classes, learns novel classes better, and produces a higher harmonic mean.The authors report results from session 2 to 11 because session 1 has no novel task.
- Sensitivity analysis: On MiniImageNet, increasing τ helps address forgetting up to τ = 2, while 3 ≤ N ≤ 5 empirically combines global and local semantic information effectively.Very small N merges super-categories, whereas N ≥7 reduces shared local information; τ values above 2 reduce accuracy.
5. Conclusion
The paper proposes semantic-aware knowledge distillation for FSCIL, using class semantics and attention-based visual embeddings to support learning without forgetting. Experiments on three datasets show improved performance over existing FSCIL methods.
- The proposed method uses word vectors as auxiliary class-semantic information, enabling knowledge distillation for FSCIL despite limited novel-class data.It is designed to support learning without forgetting when novel classes have few training examples.
- An attention mechanism combines multiple embedding representations of visual data to describe novel classes and improve generalization.
- Experiments on MiniImageNet, CUB200, and CIFAR100 demonstrate that class semantics can support knowledge distillation in FSCIL.
- The method outperforms existing state-of-the-art FSCIL approaches by a large margin.