PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Few-Shot Class-Incremental Learning

Songsong Tian, Lusi Li, Weijun Li, Hang Ran, Xin Ning, Prayag Tiwari

arXiv:2304.08130v2cs.CV

TL;DR

FSCIL must learn new classes from few labeled samples while retaining earlier knowledge, but limited data creates overfitting and catastrophic forgetting challenges. This paper surveys FSCIL research, organizes methods into five families, evaluates benchmark performance, and reviews applications. It reports dataset- and method-dependent benchmark leaders and identifies balancing performance, scalability, efficiency, and complexity as an open challenge.

  • Problem

    FSCIL requires learning new tasks from few labeled samples without forgetting prior knowledge, while limited data also creates overfitting risks.

  • Method

    The paper comprehensively surveys FSCIL approaches, proposes a five-family taxonomy, evaluates benchmark performance, and reviews applications.

  • Results

    Benchmark leaders vary by dataset: NC-FSCIL reaches 67.50% on CIFAR-100, FeSSSS reaches 68.24% on MiniImageNet, and DSN with AA reaches 71.02% on CUB-200.

  • Takeaways & Limitations

    The survey identifies integrating methods to balance performance, scalability, efficiency, and complexity as a direction for future FSCIL research.

  • Takeaways & Limitations

    Prototype-based methods face a semantic gap between few-shot class prototypes and the real data distribution, limiting accuracy improvement.

Abstract

from arXiv · show

Large deep learning models are impressive, but they struggle when real-time data is not available. Few-shot class-incremental learning (FSCIL) poses a significant challenge for deep neural networks to learn new tasks from just a few labeled samples without forgetting the previously learned ones. This setup easily leads to catastrophic forgetting and overfitting problems, severely affecting model performance. Studying FSCIL helps overcome deep learning model limitations on data volume and acquisition time, while improving practicality and adaptability of machine learning models. This paper provides a comprehensive survey on FSCIL. Unlike previous surveys, we aim to synthesize few-shot learning and incremental learning, focusing on introducing FSCIL from two perspectives, while reviewing over 30 theoretical research studies and more than 20 applied research studies. From the theoretical perspective, we provide a novel categorization approach that divides the field into five subcategories, including traditional machine learning methods, meta-learning based methods, feature and feature space-based methods, replay-based methods, and dynamic network structure-based methods. We also evaluate the performance of recent theoretical research on benchmark datasets of FSCIL. From the application perspective, FSCIL has achieved impressive achievements in various fields of computer vision such as image classification, object detection, and image segmentation, as well as in natural language processing and graph. We summarize the important applications. Finally, we point out potential future research directions, including applications, problem setups, and theory development. Overall, this paper offers a comprehensive analysis of the latest advances in FSCIL from a methodological, performance, and application perspective.

1. Introduction

Few-shot class-incremental learning addresses learning new categories from limited data while preserving prior knowledge, a setting threatened by catastrophic forgetting, overfitting, and class imbalance. This survey reviews FSCIL foundations, methods, benchmark performance, applications, and future challenges.

  • FSCIL requires models to learn new tasks incrementally from few labeled samples without forgetting previously learned ones.
  • Fine-tuning on new data can cause catastrophic forgetting and overfitting, reflecting the stability-plasticity dilemma and unreliable empirical risk minimization.
  • Limited and imbalanced samples make FSCIL models favor old classes and hinder learning of new categories.
  • Existing reviews commonly examine few-shot learning or incremental learning separately, leaving FSCIL comparatively underexplored.
  • The survey analyzes fundamental and applied FSCIL research, proposes a technique-based taxonomy, evaluates benchmark performance, discusses applications, and identifies future challenges.

2. Problem definition

FSCIL combines few-shot learning with class-incremental learning: models receive disjoint class sessions, learn from limited novel-class data, and must retain performance across all seen classes. The section situates this setup among related learning paradigms.

  • FSCIL consists of a base session with sufficient training data followed by incremental sessions containing limited data for new classes.
  • Training data from different sessions have disjoint label spaces, and incremental data can follow an N-way K-shot format.
  • At each incremental session, the model uses new data and the previous model while minimizing loss over seen classes and testing across their combined label space.
  • Few-shot Learning: Few-shot learning uses a small training set, commonly with K examples from each of N classes, which creates a large model-bias gap between expected and empirical risk.
  • Incremental Learning: Incremental learning retains and integrates prior knowledge while processing continuously arriving information, with Class-IL considered its most difficult scenario.
  • Related settings: Unlike FSCIL, generalized few-shot incremental learning aims to maintain old- and new-class performance from a pretrained model, while this review focuses on FSCIL.

3. Methods for few-shot learning

Few-shot learning methods address limited annotated data through data augmentation, metric learning, specialized architectures, and optimization strategies. These approaches improve data diversity, similarity-based classification, task adaptation, or few-shot convergence.

  • Few-shot learning surveys commonly categorize methods as data augmentation, metric-based, model-based, and optimization-based approaches.
  • Data augmentation: Data augmentation increases existing-data diversity rather than collecting new data, reducing overfitting risk and improving generalization.
  • Metric-based methods: Metric-based methods classify samples in embedding space by measuring support-query similarity or distance, including Siamese, Matching, and Prototypical Networks.
  • Model-based methods: Model-based methods use specialized architectures, such as external-memory networks, to store class information or adapt inductive biases across tasks.
  • Optimization-based methods: Optimization-based methods modify losses, regularization, or optimization algorithms so models converge quickly on few-shot data without overfitting.

4. Few-shot class-incremental learning: taxonomy

The survey organizes FSCIL methods into five families because broader metric-based and fine-tuning-based classifications do not adequately distinguish many studies. It reviews approaches spanning optimization, statistical distributions, and other techniques for learning novel classes while retaining prior knowledge.

  • Earlier work divided FSCIL into metric-based and fine-tuning-based methods, with fine-tuning approaches termed Base Classes Pretraining and Novel Classes Fine-tuning (BPNF).BPNF pre-trains on data-rich base classes and fine-tunes on novel classes during the incremental phase.
  • 33 advanced FSCIL studies are categorized into traditional machine learning, meta learning-based, feature and feature space-based, replay-based, and dynamic network structure-based methods.The taxonomy is based on each method’s key point or technique.
  • The survey distinguishes methods by underlying technique rather than experimental settings, so methods using meta-learning ideas are not necessarily classified as meta learning-based.Classification depends on the method’s key technique, not merely the presence of meta-learning-like settings.
  • Semi-supervised and self-supervised methods supplement scarce labeled novel-class data, including a setting with 50 unlabeled examples per incremental session.Unlabeled data are combined with labeled data, while self-supervision is used to address overfitting and catastrophic forgetting.
  • Statistical distribution: Statistical-distribution approaches such as GP-Tree and LDC achieve good FSCIL results, but their modeling processes are complex.LDC uses Gaussian sampling and cyclic updates to restore old-class and calibrate new-class distributions with low memory consumption.
  • Function optimization: Searching for flat local minima during base training and fine-tuning within the resulting flat region substantially reduces catastrophic forgetting.The approach treats flat minima as robust and naturally regularizing regions that can improve generalization and reduce overfitting.

4.2. Meta learning-based methods

Meta learning-based FSCIL methods are divided into prototype learning and meta-process approaches. They address limited-data adaptation through prototypes, feature-space structure, parameterized representations, and objectives that jointly handle new and old knowledge.

  • Meta learning-based FSCIL methods are divided into prototype learning-based and meta process-based methods.This categorization separates prototype-centered representations from broader meta-learning procedures.
  • Prototype learning: Prototype learning represents classes with exemplars and classifies new samples using similarity to those prototypes, requiring less labeled data than traditional supervised learning.The class prototype is computed from embedded samples, with fθ denoting the embedding network.
  • Prototype learning: Methods address prototype limitations through class-structure regularization, quantized reference vectors, dynamically growing memory, enhanced prototype representations, and neural-collapse-inspired prototype alignment.These techniques regulate embedding-space class distributions, preserve information, or correct feature-classifier misalignment.
  • Prototype learning: Prototype-based FSCIL faces prototype-representation failures from concept or distribution drift and conflicts when later-task classes differ conceptually from earlier classes.These problems can make prototype distance measurements less effective and reduce classification accuracy.
  • Prototype learning: The semantic gap between few-shot class prototypes and real data distributions remains a major obstacle to improving prototype-based accuracy.This limitation persists despite the algorithmic conciseness of the reviewed approaches.
  • Meta process: MetaFSCIL converts adapting to new knowledge and retaining old knowledge into a meta-objective using sampled sequences of incremental tasks.It additionally uses bi-directional guided modulation to adapt to new knowledge.
  • Meta process: Large-margin classification can improve base-class performance while reducing novel-class performance, motivating the boundary-based CLOM framework.The survey terms this phenomenon class-level overfitting and attributes it to easily satisfied shared or class-specific pattern constraints.

4.3. Feature and feature space-based methods

Feature and feature space-based methods reshape representations or reserve feature-space capacity to improve adaptation and retention in FSCIL. The reviewed approaches include feature decoupling, subspaces, virtual classes, and prospective learning.

  • Feature decoupling: Feature decoupling separates representations so models can emphasize relevant information, with low-frequency components helping preserve old knowledge.A discrete cosine transform and frequency-aware regularization are used to support inter-space learning.
  • Subspace representation: Subspace representation maps data into lower-dimensional spaces and projects new-class data into subspaces formed by base or old-class features.Mixtures of subspaces describe visual and semantic domain distributions.
  • Fantasy space: Fantasy space introduces placeholders for unseen classes through discrete transformations of original classes to improve base-class separation and novel-class generalization.The model learns to recognize and contrast virtual classes in the constructed space.
  • Compatibility: FSCIL’s resistance to forgetting is framed as backward compatibility, meaning new-session training should retain compatibility with previously learned information.This framing connects incremental learning to software-update compatibility concepts.
  • Compatibility: Prospective Learning trains on base data for forward compatibility so the model can better handle incremental few-shot novel classes.The definition emphasizes preparation for future novel-class learning.
  • Compatibility: FACT reserves virtual prototypes in feature space, optimizes them for compact intra-class structure, and uses instance mixing to generate virtual instances.The reserved space is intended to make the model scalable for upcoming classes.

4.4. Replay-based methods

Replay-based FSCIL methods revisit information from earlier tasks while learning the current task. They use either stored exemplars or generated samples, sometimes combined with parameter constraints, classifier calibration, distillation, or privacy-oriented synthesis.

  • Replay-based methods use episodic memory to replay previous-task information while updating the model on the current task.Direct replay stores old examples, whereas generative replay models and generates old-data distributions.
  • The replay loss averages prediction loss over the union of current-task data D(t) and memory M(t).This formulation combines present data with replayed information during fine-tuning.
  • A three-stage framework separately trains on base and novel classes, constrains model parameters, then replays stored samples to calibrate performance across all classes.The final stage covers both base and novel classes.
  • Knowledge-distillation replay stores old-class exemplars and adds a distillation loss to transfer and preserve old knowledge.The replay mechanism is paired with an additional objective beyond ordinary supervised fine-tuning.
  • Data-free replay synthesizes old samples to address privacy concerns from storing real data, while entropy regularization encourages uncertain examples near decision boundaries.FSIL-GAN is proposed because traditional generative replay from class-incremental learning cannot be directly applied to FSCIL.

4.5. Dynamic network structure-based methods

Dynamic network structure-based methods adapt architectures or relationships as FSCIL data evolves, aiming to learn new classes while retaining prior knowledge. The reviewed approaches include topology growth, graph-based classifiers, expansion-compression networks, and soft subnetworks.

  • Dynamic architectures adjust network structure during runtime according to input features, supporting flexible adaptation in incremental learning.The survey associates these architectures with generalization and reduced overfitting risks.
  • TOPIC uses a neural gas network whose topology grows for new samples while stability constraints preserve representations of old categories.Its topology represents the feature space formed by different categories.
  • CEC decouples backbone embedding learning from classifier learning and uses a graph attention model to support a continually evolving classifier.The decoupling is intended to prevent forgetting in the backbone.
  • LEC-Net selectively expands nodes to strengthen feature representations and reduce feature drift through model regularization.
  • DSN adaptively expands and compresses network capacity, enriching representations during each incremental task while adjusting feature space using old-class distributions.
  • Masking-based methods construct soft subnetworks with non-binary masks, updating only minor subnetwork parameters during incremental learning to balance forgetting and overfitting.

4.6. Methods summary

The survey compares five FSCIL method families and identifies distinct strengths, constraints, and unresolved trade-offs. It highlights limitations involving distribution assumptions, storage and replay quality, statistical modeling complexity, and overall system design.

  • Traditional methods improve interpretability through statistical distribution or function-optimization perspectives, but statistical distribution modeling remains difficult.
  • Meta-learning methods seek flexibility and adaptation, but depend heavily on meta-training data and may degrade when incremental tasks differ from base-class distributions.
  • Feature-based methods focus on learning robust, efficient representations and are considered promising for handling unseen samples.
  • Replay methods directly address catastrophic forgetting, while direct replay faces storage, sample-selection, and privacy constraints.Generative replay reduces some of these constraints but retains training-complexity and data-quality challenges.
  • Dynamic network methods adjust structures or inter-class relationships to learn new knowledge while retaining old knowledge.
  • FSCIL remains an open challenge requiring a balance among performance, scalability, efficiency, and complexity.

5. Model performance

The survey evaluates FSCIL methods across three benchmark image-classification datasets using session accuracy, average accuracy, and performance dropping rate. Results vary substantially by dataset and method, with different approaches leading on CIFAR-100, MiniImageNet, and CUB-200.

  • Experimental setup: The evaluation compares 22 methods from five families using reported or processed original-paper data because many implementations are unavailable.The authors note that most studies follow the standards established by Tao et al. (2020).
  • Datasets: The benchmarks are CIFAR-100, MiniImageNet, and CUB-200, with CUB-200 providing more sessions and incremental classes for sensitivity comparisons.
  • Metrics: The compared metrics are session accuracy, average accuracy across sessions, and performance dropping rate.Performance dropping rate measures the absolute accuracy drop from the base session to the last session.
  • Benchmark results: 67.50%: NC-FSCIL achieves the strongest reported performance on CIFAR-100 among the compared methods.
  • Benchmark results: 68.24%: FeSSSS achieves the strongest reported performance on MiniImageNet and exhibits lower knowledge forgetting than NC-FSCIL.
  • Benchmark results: 71.02%: DSN is the only method exceeding 70% average accuracy on CUB-200, indicating stronger fine-grained category discrimination.
  • Session comparison: Most methods reach 70% to 80% accuracy on CUB-200 base classes, while F2M, DSN, and NC-FSCIL are among the few exceeding 80%.The session comparison also identifies F2M and DSN as retaining high performance advantages.

6. Research on few-shot incremental learning applications

FSCIL applications extend across computer vision, natural language processing, and graph domains, covering classification, detection, segmentation, and other specialized tasks. These studies adapt methods such as meta-learning, feature registration, knowledge distillation, replay, and prototype-based learning to domain-specific incremental settings.

  • Cross-domain applications: FSCIL applications span computer vision, natural language processing, and graph technology rather than remaining limited to general image classification.The surveyed applications include image classification, object detection, image segmentation, NLP tasks, and graph learning.
  • Image classification: MetaSearch uses multipooling-based feature extraction to capture subtle differences among fine-grained product categories for incremental product search.The method generates features distinguishing novel categories in shopping and checkout scenarios.
  • Object detection: Few-shot incremental object detection includes road-object systems that use only 10 annotated instances per new class.DualFusion combines base-class detection, novel-class detection, and a fusion network, but requires retaining all novel few-shot data.
  • Image segmentation: Bayesian uncertainty estimation modifies classification and bounding-box heads for incremental few-shot instance segmentation, outperforming iMTFA on COCO.The surveyed work improves performance but does not explain why its bounding-box uncertainty estimate surpasses Gaussian-based estimation.
  • NLP and multimodal applications: NLP applications address intent recognition, continual few-shot relation learning, named entity recognition, and label-to-image translation through contrastive, regularization, replay, and adaptive-filter methods.These approaches target forgetting and incremental learning in text and multimodal settings.
  • Graph applications: Graph FSCIL methods use hierarchical attention, metric-space prototypes, knowledge distillation, and biased sampling to address class imbalance and forgetting.HAG-Meta and Geometer nevertheless cannot handle dynamic graph structures.

7. Future works

Future FSCIL research should improve efficiency and evaluation, adopt more realistic variable-shot settings, and broaden applications beyond single-scene domains. The survey also identifies unified theory and newer architectures as important directions.

  • Theories: Future methods should jointly consider performance and complexity rather than optimizing accuracy without accounting for resource requirements.The survey also calls for stronger testing standards across multiple tasks and the base dataset.
  • Theories: Average accuracy overlooks imbalanced base and novel data, while relative performance dropping rate and harmonic accuracy capture performance more comprehensively.The survey notes that conventional performance dropping rate ignores intermediate incremental sessions.
  • Applications: Future work may explore Vision Transformers in FSCIL alongside theories, realistic settings, and broader interdisciplinary applications.The survey cites ViT-based FSCIL as an example of a promising architectural direction.
  • FSCIL settings: More realistic FSCIL settings should allow variable numbers of new classes and shots, imbalanced novel classes, and fewer base classes.The survey also suggests combining few-shot learning with task-incremental and domain-incremental learning.
  • Applications: Applications in audio, radar, intrusion detection, and medical time-series classification remain limited by single-scene settings and lack scalability.The survey identifies a unified framework for complex and multimodal tasks as a future direction.

8. Conclusion

The survey organizes FSCIL methods into five families, evaluates representative approaches, and reviews applications across deep learning fields. It concludes that FSCIL remains underexplored, with future work needed on integrated methods, applications, and theory.

  • Conclusion: The survey categorizes FSCIL into traditional machine learning, meta-learning, feature and feature-space, replay-based, and dynamic-network methods.It also discusses their performance and applications across multiple deep learning fields.
  • Conclusion: FSCIL remains underexplored, and the survey identifies further research on balancing performance, scalability, efficiency, and complexity.The conclusion also notes that some theoretical derivations were not extensively introduced because of space limitations.
Loading 2304.08130v2…