Source-linked AI summary
Learning with Fantasy: Semantic-Aware Virtual Contrastive Constraint for Few-Shot Class-Incremental Learning
Zeyin Song, Yifan Zhao, Yujun Shi, Peixi Peng, Li Yuan, Yonghong Tian
TL;DR
FSCIL seeks to learn new classes from few examples without forgetting old ones, yet the common base-training and frozen-backbone approach can leave poor separation for novel-class generalization. SAVC introduces transformed virtual classes in a fantasy space; it reports state-of-the-art performance across three FSCIL benchmarks.
Problem
In FSCIL, cross-entropy base training followed by backbone freezing can leave base classes poorly separated, limiting generalization to novel classes with few examples.
Source: §1 · pp. 1–2Method
SAVC uses pre-defined transformations to create virtual classes in a fantasy space and learns to recognize and contrast across its semantic extensions.
Source: §1 · p. 2Results
Takeaways & Limitations
Virtual classes hold feature space for future updates and enable multi-semantic aggregated inference, improving classification accuracy.
Source: §5 · p. 8The described setting provides sufficient data only in the base session and uses N-way K-shot data in later sessions.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Few-shot class-incremental learning (FSCIL) aims at learning to classify new classes continually from limited samples without forgetting the old classes. The mainstream framework tackling FSCIL is first to adopt the cross-entropy (CE) loss for training at the base session, then freeze the feature extractor to adapt to new classes. However, in this work, we find that the CE loss is not ideal for the base session training as it suffers poor class separation in terms of representations, which further degrades generalization to novel classes. One tempting method to mitigate this problem is to apply an additional naive supervised contrastive learning (SCL) in the base session. Unfortunately, we find that although SCL can create a slightly better representation separation among different base classes, it still struggles to separate base classes and new classes. Inspired by the observations made, we propose Semantic-Aware Virtual Contrastive model (SAVC), a novel method that facilitates separation between new classes and base classes by introducing virtual classes to SCL. These virtual classes, which are generated via pre-defined transformations, not only act as placeholders for unseen classes in the representation space, but also provide diverse semantic information. By learning to recognize and contrast in the fantasy space fostered by virtual classes, our SAVC significantly boosts base class separation and novel class generalization, achieving new state-of-the-art performance on the three widely-used FSCIL benchmark datasets. Code is available at: https://github.com/zysong0113/SAVC.
1. Introduction
FSCIL must add classes from few examples without forgetting old ones, but the incremental-frozen framework leaves open what makes a base-session model generalize to new classes. SAVC introduces virtual classes in a fantasy space to improve class separation and reports state-of-the-art performance across three FSCIL benchmarks.
1. Introduction
- The mainstream FSCIL framework trains with cross-entropy at the base session, then freezes the backbone to adapt to new classes.Source: §1 · p. 2
- CE provides weaker base-class separation than SCL, while SCL's slight improvement comes with reduced inter-class distance, leaving inadequate room for new classes.Source: §1 · p. 2
- Virtual classes with varied semantics can fill unallocated representation space despite future classes being unavailable during base training.Source: §1 · p. 2
- SAVC uses pre-defined transformations to build a finer-grained fantasy space, decreasing intra-class distance and increasing inter-class distance versus CE and SCL.Source: §1 · p. 2
- SAVC achieves state-of-the-art performance on CIFAR100, miniImageNet, and CUB200 FSCIL benchmarks.Source: §1 · p. 2
2. Related Work
Related work spans few-shot adaptation, class-incremental learning, FSCIL, and self-supervised representation learning. The paper distinguishes FSCIL from approaches that assume many training examples or do not preserve performance on base classes.
2. Related Work
- Few-shot learning adapts to unseen classes with few samples, but does not directly address retaining base-class performance as FSCIL requires.Source: §2 · p. 2
- Class-incremental methods use rehearsal, parameter regularization, or network expansion, but rely on many training examples and are unsuitable for FSCIL.Source: §2 · p. 2
- FSCIL methods either update the backbone during incremental sessions or freeze it, with frozen-backbone work addressing stability and plasticity in several ways.Source: §2 · pp. 2–3
- Self-supervised learning uses pretext tasks, while contrastive methods aggregate positive pairs and exclude negatives; supervised contrastive learning uses labels to define positives.Source: §2 · p. 3
3. Methodology
SAVC augments the incremental-frozen FSCIL framework with transformation-generated virtual classes, training classification and contrastive branches in a fantasy space and aggregating predictions across transformed views. Its design targets better base-class separation and novel-class generalization.
3.1. Preliminary
- FSCIL provides sufficient base-session data, then only a few N-way K-shot samples per class; the incremental-frozen baseline trains with CE, freezes the backbone, and extends class prototypes.Inference uses nearest class mean over the encountered classes.Source: §3.1 · p. 3
3.3. Semantic-Aware Contrastive Class Fantasy
- SAVC increases inter-class distance and decreases intra-class distance, yielding the best base-class separation and novel-class generalization among the compared methods.The comparison is against CE and CE+SCL.Source: Figure 2 · p. 3, §3.3 · p. 5
3.2. Base Class Separation Facilitates Novel Class Generalization
- SCL modestly improves mutual base–novel separation but does not separate novel representations from old-class distributions, despite improving overall class-separation degree and accuracy.The section measures prototype inter-class distance, within-class compactness, and mutual base–novel separation.Source: §3.2 · pp. 4–5
3.3. Semantic-Aware Contrastive Class Fantasy
- Virtual classes generated by predefined transformations expand each original class into M fine-grained labels, serving as placeholders for unseen classes and providers of diverse semantics.Classification and contrastive learning are performed across the transformed samples in fantasy space.Source: §3.3 · p. 5
- At inference, transformed views are matched to transformation-specific prototype subsets, whose predictions are aggregated and projected into the original label space.This is the multi-semantic aggregated inference procedure.Source: Figure 3 · p. 4, §3.3 · p. 5
3.4. Discussion
- Multi-Crop adds N small query-only crops overlapping global crops above a threshold, with separate global and local contrastive losses in the training objective.The small crops are not fed to the key encoder.Source: §3.4 · p. 6
- During incremental sessions, SAVC finetunes the pretrained model and contrasts novel queries with queued base-session key embeddings, which helps push novel samples away from old-class distributions.Source: §3.4 · p. 6
4. Experiments
Across three FSCIL benchmarks, SAVC achieves the highest average accuracy over all sessions, with the largest reported final-accuracy advantage over FACT on miniImageNet. Ablations, visualizations, and further analyses examine the components and settings associated with its performance.
4.1. Experimental Setup
- The experiments use CIFAR100, miniImageNet, and CUB200, with dataset-specific backbones, transformation sets, and α = 0.2 and β = 0.8.The setup uses ResNet20 for CIFAR100 and ResNet18 for miniImageNet and CUB200; it uses 12 augmentations for miniImageNet and 2-fold rotations for the other datasets.Source: §4.1 · p. 6, Table 4 · p. 7
4.3. Ablation study
- On miniImageNet, fantasy virtual classes, multi-crop, and incremental finetuning yield reported gains of 8.98%, 10.26%, and 11.57%, while SCL alone gains 1.56%.The authors describe virtual classes as preventing overlap between old and new distributions; the reported gains are relative to the CE baseline in the last session.Source: Table 3 · p. 7, §4.3 · p. 7
4.2. Comparison with State of The Arts
- 53.12% on CIFAR100, 62.50% on CUB200, and 57.11% on miniImageNet are SAVC’s final accuracies, exceeding FACT by 1.02%, 5.56%, and 6.62%, respectively.SAVC consistently achieves the highest average accuracy over all sessions on the three datasets.Source: §4.2 · p. 7
4.4. Visualization
- On CUB200, SAVC’s last-session confusion matrix shows better performance on both base and new classes than CE’s, whose new-class performance is poor.The figure’s bright diagonal and dim background indicate higher classification accuracy.Source: Figure 5 · p. 7, §4.4 · p. 7
- On CIFAR100, SAVC’s embeddings separate base classes and incorporate novel classes with the least overlap, while CE and SCL leave separation or confusion problems.The visualization samples six base classes and three novel classes; its caption identifies classes 0–5 as base and 6–8 as novel.Source: §4.4 · pp. 7–8, Figure 7 · p. 8
4.5. Further Analysis
- Complex transformations help miniImageNet but hurt class separation on CUB200 and CIFAR100; the authors suggest fine-grained images and low resolution may explain the difference.The authors hypothesize that overly fine grains hinder learning on CUB200, while CIFAR100’s small images make original and virtual classes difficult to distinguish.Source: §4.5 · p. 8, Figure 6 · p. 8
- On miniImageNet, finetuning fewer and deeper layers gives higher accuracy, while finetuning too many layers harms prior knowledge and performs worse.The authors infer that shallow-layer features are shared between classes, so updating higher layers is sufficient for adaptation to new classes.Source: §4.5 · p. 8, Figure 6 · p. 8
- On miniImageNet, the method is robust to α and β choices and performs best at α = 0.2 and β = 0.8; those values are fixed across datasets.The coefficients control the impact of global and local supervised contrastive loss.Source: §4.5 · p. 8, Figure 6 · p. 8
5. Conclusion
SAVC addresses FSCIL by adding virtual classes to the incremental-frozen framework, where CE and SCL alone fail to separate base and new classes well. The paper reports state-of-the-art performance on three popular FSCIL benchmarks.
5. Conclusion
- CE and SCL fail to separate base classes and new classes well in the incremental-frozen framework.The conclusion identifies this as the motivation for introducing more semantic knowledge through virtual classes.Source: §5 · p. 8
- SAVC introduces virtual classes that provide feature space for future updates and enable multi-semantic aggregated inference.The method learns to recognize and contrast in the fantasy space.Source: §5 · p. 8
- SAVC maintains separation of all classes and achieves state-of-the-art performance on three popular FSCIL benchmarks.Source: §5 · p. 8