PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks

Da-Wei Zhou, Han-Jia Ye, Liang Ma, Di Xie, Shiliang Pu, De-Chuan Zhan

arXiv:2203.17030v2cs.CVcs.LG

TL;DR

FSCIL asks models to learn new classes from limited examples without forgetting old classes, a challenge involving both overfitting and catastrophic forgetting. LIMIT meta-trains on simulated multi-phase tasks and uses transformer-based calibration between old classifiers and new prototypes. The paper reports state-of-the-art performance on three benchmark datasets and ImageNet ILSVRC2012.

  • Problem

    FSCIL must learn sequentially arriving classes from limited instances while preserving old-class discrimination, despite overfitting and catastrophic forgetting.

  • Method

    LIMIT meta-trains on sampled fake-FSCIL tasks and uses a transformer meta-calibration module to align old classifiers with few-shot new prototypes.

  • Results

    LIMIT achieves state-of-the-art performance on three benchmark datasets and ImageNet ILSVRC2012.

  • Takeaways & Limitations

    Multi-phase simulated-task training prepares LIMIT for incoming few-shot incremental sessions while its calibration module maintains old-class discriminability during new-class learning.

  • Takeaways & Limitations

    The calibration module uses a transformer set-to-set function, while alternative set functions are left for future work.

Abstract

from arXiv · show

New classes arise frequently in our ever-changing world, e.g., emerging topics in social media and new types of products in e-commerce. A model should recognize new classes and meanwhile maintain discriminability over old classes. Under severe circumstances, only limited novel instances are available to incrementally update the model. The task of recognizing few-shot new classes without forgetting old classes is called few-shot class-incremental learning (FSCIL). In this work, we propose a new paradigm for FSCIL based on meta-learning by LearnIng Multi-phase Incremental Tasks (LIMIT), which synthesizes fake FSCIL tasks from the base dataset. The data format of fake tasks is consistent with the `real' incremental tasks, and we can build a generalizable feature space for the unseen tasks through meta-learning. Besides, LIMIT also constructs a calibration module based on transformer, which calibrates the old class classifiers and new class prototypes into the same scale and fills in the semantic gap. The calibration module also adaptively contextualizes the instance-specific embedding with a set-to-set function. LIMIT efficiently adapts to new classes and meanwhile resists forgetting over old classes. Experiments on three benchmark datasets (CIFAR100, miniImageNet, and CUB200) and large-scale dataset, i.e., ImageNet ILSVRC2012 validate that LIMIT achieves state-of-the-art performance.

1 INTRODUCTION

Few-shot class-incremental learning must absorb sequentially arriving classes from limited examples while preserving performance on previously learned classes. LIMIT addresses this through multi-phase meta-training on simulated tasks and transformer-based calibration between old classifiers and new prototypes.

  • Motivation: FSCIL requires learning sequentially arriving classes from limited data while retaining discriminability over all previously seen classes.The setting combines catastrophic forgetting with overfitting on few-shot incremental inputs.
  • Motivation: Directly applying conventional CIL methods performs poorly because few-shot updates can overfit and damage pretrained feature embeddings.Damaged embeddings lose characteristics of former classes, worsening forgetting.
  • LIMIT: LIMIT samples diverse fake-FSCIL tasks from the base dataset to cultivate long-term incremental-learning ability and generalizable features.The simulated tasks follow the incremental structure of real FSCIL tasks.
  • LIMIT: LIMIT uses a transformer set-to-set function to calibrate many-shot old classifiers and few-shot new prototypes, reducing prediction bias from their information gap.The module also adapts embeddings with instance-specific contextual information.

2 RELATED WORK

Related work spans class-incremental learning, few-shot learning, FSCIL, generalized few-shot learning, and transformer-based visual modeling. FSCIL extends few-shot adaptation to repeated incremental stages, making it more challenging than single-stage generalized few-shot learning.

  • 2.1 Class-Incremental Learning: Class-incremental learning methods either consolidate old knowledge without exemplars or rehearse representative old instances during updates.Examples include regularization and dynamic structures in non-exemplar methods, and replay or distillation in exemplar-based methods.
  • 2.2 Few-Shot Learning (FSL): Few-shot learning extracts inductive bias from base classes to generalize to unseen classes through optimization-based or metric-based approaches.Optimization-based methods target rapid adaptation with few examples, while metric-based methods learn distances between support and query instances.
  • 2.3 Few-Shot Class-Incremental Learning: FSCIL addresses few-shot inputs under class-incremental learning, with prior methods preserving feature topology, refining prototypes, or evolving classifiers.The cited methods include TOPIC, Exemplar Relation Graph, FSLL, semantic-aware distillation, self-promoted prototype refinement, and CEC.
  • 2.4 Generalized Few-Shot Learning: Generalized few-shot learning also combines old and new classes with limited new data, but typically contains only one incremental stage.That single-stage setting is less challenging than FSCIL's repeated incremental process.
  • 2.5 Transformer: Transformers have been adapted from natural-language processing to computer-vision tasks including detection, segmentation, and video understanding.The paper adopts transformer modeling for its FSCIL calibration module.

3 FROM OLD CLASSES TO NEW CLASSES

FSCIL extends a well-trained base model with sequential, few-shot class sets while preserving performance across all seen classes. The section contrasts direct CIL and prototype-based approaches, highlighting forgetting and overfitting constraints that motivate LIMIT’s synthetic-task and calibration strategy.

  • Few-Shot Class-Incremental Learning: FSCIL receives a sufficient-data base session followed by sequential N-way K-shot tasks containing disjoint new classes.At task b, only the current dataset Db is accessible, while performance must be maintained over all seen classes.
  • Few-Shot Class-Incremental Learning: The learning objective minimizes expected risk on each incremental task and across the cumulative sequence of sessions.The new model is built from the current dataset and the preceding embedding and classifier while covering all seen classes.
  • Limitations of Existing Approaches: Directly applying CIL methods can overfit few-shot inputs, damage pretrained embeddings, and worsen catastrophic forgetting.Knowledge distillation trades off learning new classes and retaining old ones with sufficient data, but limited FSCIL data can quickly destroy the pre-learned embedding.
  • LIMIT: LIMIT samples fake incremental tasks from the base dataset and uses meta-calibration to align old classifiers, new prototypes, and query embeddings.The calibration process uses a set-to-set function and produces instance-specific embeddings before computing logits by inner product.
  • Prototype-Based Adaptation: Prototypical networks fix the embedding and represent each new class by the average embedding of its few-shot instances.This approach resists overfitting by adapting new classes without changing the pretrained embedding, but captures only static characteristics.

4 LEARNING LIMIT FOR FSCIL

LIMIT meta-learns FSCIL by sampling multi-phase fake-incremental tasks from the base dataset, then calibrates old classifiers and new prototypes during adaptation. This training format matches real incremental sessions and supports generalization to future classes.

  • Learning Multi-Phase Incremental Tasks: LIMIT samples multi-phase fake-incremental tasks from base classes to mimic the structure of real FSCIL sessions.The sampled tasks contain sequential support and query sets, with each phase adding new few-shot classes while evaluating over previously seen classes.
  • Learning Multi-Phase Incremental Tasks: The fake-task objective evaluates the learning strategy across all phases and seen classes, enabling meta-learning of a generalizable feature space.Query sets include old and new classes, while the empirical risk reflects the dynamic multi-phase FSCIL process.
  • Learning Multi-Phase Incremental Tasks: During each fake phase, new-class classifiers are replaced with prototypes, while classifiers for older classes are retained from the previous phase.For the first phase, the old classifiers come from the pretrained base model; subsequent phases update the classifier set incrementally.
  • Meta-Calibration Module: Meta-calibration addresses the semantic gap between old classifiers and new prototypes by adapting their representations and logits with a transformer-based set-to-set function.The module also produces instance-specific embeddings that emphasize query-dependent discriminative information.
  • Training and Inference: Meta-training repeatedly samples fake task sequences, calibrates prototype-augmented classifiers, and optimizes the resulting predictions before real FSCIL inference.At inference, new-class prototypes augment the current classifier, and the same calibration process is applied to test instances.
  • Discussion: Multi-phase fake-task sampling is presented as more suitable than one-phase training because real FSCIL involves multiple incremental sessions.The paper treats one-phase training as a degradation version and reports experimental verification of this design choice.

5 EXPERIMENT

LIMIT is evaluated against state-of-the-art FSCIL methods on three benchmark datasets and large-scale ImageNet variants, using incremental accuracy and performance dropping rate to assess recognition and forgetting.

  • Datasets and Protocols: Experiments cover CIFAR100, CUB200, miniImageNet, ImageNet100, and ImageNet1000 under few-shot class-incremental settings.The protocols include 5-way 5-shot, 10-way 5-shot, and 50-way 5-shot incremental sessions across the datasets.
  • Benchmark Results: 5.13 higher last accuracy and 5.09 lower performance dropping rate than the runner-up are reported on CUB200.The table compares LIMIT with state-of-the-art methods and also reports a LIMIT variant using CEC’s data augmentation.
  • Compared Methods: LIMIT is compared with classical class-incremental methods, current FSCIL algorithms, and a finetuning baseline using a shared network backbone.Compared FSCIL methods include TOPIC, Decoupled-DeepEMD/Cosine, and CEC.
  • Benchmark Results: 1.56 higher last accuracy and 1.24 lower performance dropping rate than the runner-up are reported for one evaluated dataset.Performance dropping rate is defined as the accuracy decrease from the base session to the last incremental session.

5.2 Ablation Study

The ablation study separates prototype initialization, transformer-based meta-calibration, and fake-task training, showing progressively stronger performance from the combined LIMIT components and multi-phase training.

  • Component Analysis: Directly optimizing with few-shot images without LIMIT components suffers severe catastrophic forgetting.This is identified as the first ablation configuration in Table 1.
  • Component Analysis: Prototype initialization relieves overfitting and forgetting, while meta-calibration further calibrates relationships between old and new classes and improves performance.The transformer is used for calibration rather than as the embedding; the improvement is attributed to learned calibration information during meta-learning.
  • Fake-Task Training: Multi-phase meta-training prepares the model for multi-phase incremental training and helps obtain a more generalizable feature space.The ablation compares one-phase and multi-phase fake-task training and reports substantial improvement over the ProtoNet baseline.

5.3 Benchmark Comparison

LIMIT consistently outperforms state-of-the-art methods across benchmark and large-scale datasets while resisting forgetting and improving new-class generalization.

  • Benchmark Results: LIMIT consistently outperforms state-of-the-art methods on CIFAR100, miniImageNet, and CUB200.It exceeds runner-up methods by 1.5%–2% on CIFAR100 and miniImageNet and by 5% on CUB200.
  • Calibration: LIMIT’s meta-calibration module encodes calibration information between old and new classes.
  • Forgetting Resistance: LIMIT consistently resists forgetting in few-shot class-incremental tasks, as shown by performance-dropping comparisons.
  • Benchmark Results: LIMIT consistently outperforms CEC on ImageNet100 and ImageNet1000, with a substantial margin across large- and small-scale datasets.
  • Confusion-Matrix Analysis: LIMIT’s confusion matrix indicates adaptation to new classes with generalizable features while stably resisting catastrophic forgetting.
  • Generalization Analysis: LIMIT improves new-class accuracy by 13% while base-class performances remain almost the same.CEC improves new-class accuracy by 5%, whereas LIMIT uses multi-phase sampling and a transformer to extract invariant information.

5.5 Analysis of Hyper-Parameters

Hyper-parameter analyses show that multi-phase training, larger fake tasks, and more new-class examples improve LIMIT’s performance and robustness.

  • Fake Way and Shot: LIMIT does not require meta-training and testing to use the same-way same-shot protocol.The analysis varies fake ways and shots independently across 25 combinations.
  • Number of Phases: More than one fake-incremental phase performs better than one phase, supporting multi-phase training for FSCIL.
  • Multiple Trials: LIMIT remains robust across different episode combinations, while iCaRL’s performance varies between task combinations.
  • Test Shot: More instances per new class provide more information and improve prototype estimation during incremental tasks.
  • Decision-Boundary Visualization: The meta-calibration module adapts prototypes and calibrates decision boundaries between old and new classes.

5.7 Visualization of Meta-Calibration Module

Meta-calibration re-ranks predictions to better balance base and incremental classes, especially when few-shot new classes are confused with base classes.

  • Visualization Setup: Before and after meta-calibration, the visualization compares top-5 prediction probabilities for original images.Base-class probabilities are brown, incremental-class probabilities are blue, and the ground-truth class has red edges.
  • Calibration Effect: For many-shot base classes, meta-calibration addresses examples that receive incorrect base-class predictions.
  • Calibration Effect: Few-shot new-class instances tend to be predicted as base classes before calibration, but meta-calibration increases their output probabilities and re-ranks predictions.

5.8 Analysis of Incremental Task Sampling, Model Efficiency and Data Augmentations

LIMIT’s task-sampling strategy improves performance, adds modest computational cost, and benefits from stronger augmentation, although augmentation can be domain-sensitive.

  • Task Sampling Strategy: LIMIT’s fake task sampling strategy improves CEC, while replacing it with CEC’s strategy degrades LIMIT.The comparison is reported on CUB200 and CIFAR100.
  • Model Efficiency: LIMIT requires only slight extra training time for meta-calibration, with running time at the same scale as other methods.
  • Data Augmentations: Both CEC and LIMIT generally benefit from AutoAug, but AutoAug reduces performance on CUB200 because its policy is optimized for ImageNet.
  • Data Augmentations: LIMIT outperforms CEC with or without AutoAug and remains competitive with CEC using weaker augmentations.
  • Data Augmentations: LIMIT benefits more from augmentation than CEC, whose rotation-based synthetic classes may conflict with AutoAug policies.

6 CONCLUSION

LIMIT simulates multi-phase FSCIL tasks from base data and uses transformer-based meta-calibration to align old classifiers with new prototypes while contextualizing embeddings. It adapts to new classes, preserves old knowledge, and consistently achieves state-of-the-art performance.

  • LIMIT samples fake incremental tasks from base data to learn generalizable features for future FSCIL tasks.
  • A transformer-based meta-calibration module aligns old classifiers and few-shot prototypes into the same scale.
  • The calibration module generates instance-specific embeddings through a set-to-set transformation for improved prediction.
  • LIMIT efficiently adapts to new classes while preserving old knowledge and consistently achieving state-of-the-art performance.

APPENDIX A USING TRANSFORMER AS SET-TO-SET FUNCTION

The appendix describes transformer self-attention as a set-to-set function that transforms classifier, prototype, and embedding sets. Attention weights produce adapted embeddings, while the module calibrates classifiers and prototypes into a common scale.

  • Self-attention serves as the meta-calibration module, transforming query elements according to their similarities with keys and associated values.
  • The transformer projects points from the relevant sets before computing their set-to-set transformation.
  • The key and value sets have equal cardinality because each key corresponds one-to-one with a value.
  • Attention measures the proximity between a query point and the list of keys, producing weights for the final embedding.
  • Dropout and layer normalization implement the further transformation applied after the fully connected projection.
  • The transformer calibrates classifiers and prototypes into the same scale and produces instance-specific embeddings.
  • In implementation, the query, key, and value sets can jointly contain classifiers and the query embedding, with bias omitted for simplification.
  • Adapted embeddings yield more proper predictions after meta-calibration.

APPENDIX B ADDITIONAL VISUALIZATION RESULTS

Additional visualizations show how meta-calibration changes top-5 predictions for base and incremental images. Without contextualization, the model can misclassify examples by focusing on irrelevant features.

  • The appendix adds six images illustrating the meta-calibration process in Figure 11.
  • Figure 11 includes two base-class examples and four incremental-class examples.
  • Without contextualization, the model may misclassify a house-finch as a toucan or a Saluki as a Walker-hound.
  • For incremental classes, the model may focus on irrelevant features and make errors such as predicting a reel as a tank.
Loading 2203.17030v2…