PaperlayerView exact revision on arXiv

Source-linked AI summary

Forward Compatible Few-Shot Class-Incremental Learning

Da-Wei Zhou, Fu-Yun Wang, Han-Jia Ye, Liang Ma, Shiliang Pu, De-Chuan Zhan

arXiv:2203.06953v1cs.CVcs.LG

TL;DR

FSCIL asks models to learn sequentially arriving classes from few-shot data without forgetting old classes, a setting where existing methods primarily pursue backward compatibility. FACT instead reserves embedding space with virtual prototypes and forecasts possible future classes using virtual instances. FACT consistently achieves state-of-the-art performance across benchmark settings, outperforming CEC by 3–5% and surpassing it on ImageNet top-1 and top-5 accuracy.

  • Problem

    FSCIL requires incorporating new classes from insufficient instances while preserving old-class discriminability, whereas existing approaches mainly address updates retrospectively through backward compatibility.

  • Method

    FACT reserves embedding space with virtual prototypes, forecasts possible future classes using virtual instances, and uses the prototypes as informative basis vectors during incremental inference.

  • Results

    FACT consistently outperforms current state-of-the-art CEC by 3–5% on benchmark datasets and surpasses CEC on all reported ImageNet top-1 and top-5 metrics.

  • Takeaways & Limitations

    FACT efficiently incorporates new knowledge while retaining forward compatibility and resisting forgetting of old classes across small- and large-scale FSCIL tasks.

  • Takeaways & Limitations

    The analysis treats [W, Pv] as a uniform classifier and does not distinguish classifier weights from virtual prototypes for simplicity.

Abstract

from arXiv · show

Novel classes frequently arise in our dynamically changing world, e.g., new users in the authentication system, and a machine learning model should recognize new classes without forgetting old ones. This scenario becomes more challenging when new class instances are insufficient, which is called few-shot class-incremental learning (FSCIL). Current methods handle incremental learning retrospectively by making the updated model similar to the old one. By contrast, we suggest learning prospectively to prepare for future updates, and propose ForwArd Compatible Training (FACT) for FSCIL. Forward compatibility requires future new classes to be easily incorporated into the current model based on the current stage data, and we seek to realize it by reserving embedding space for future new classes. In detail, we assign virtual prototypes to squeeze the embedding of known classes and reserve for new ones. Besides, we forecast possible new classes and prepare for the updating process. The virtual prototypes allow the model to accept possible updates in the future, which act as proxies scattered among embedding space to build a stronger classifier during inference. FACT efficiently incorporates new classes with forward compatibility and meanwhile resists forgetting of old ones. Extensive experiments validate FACT's state-of-the-art performance. Code is available at: https://github.com/zhoudw-zdw/CVPR22-Fact

1. Introduction

FSCIL requires sequentially incorporating classes from few-shot data while preserving old-class discriminability. FACT addresses this prospectively by reserving embedding space and forecasting possible future classes.

  • FACT: The Figure 1 scheme contrasts conventional training with base-session reservation of embedding space for future extensions.The setting contains sequential sessions with non-overlapping classes and few-shot incremental data.
  • Motivation: FSCIL extends class-incremental learning to sequentially arriving, non-overlapping classes with limited instances, while retaining old-class discriminability.Base sessions provide ample training data, whereas incremental sessions provide few-shot instances.
  • Motivation: Existing approaches emphasize backward compatibility, shifting the burden of preserving old-class performance to later updates with limited data.The paper argues that preparing for future extensions earlier is more suitable for FSCIL.
  • Forward Compatibility: Forward compatibility makes a model growable by reserving space for future class embeddings and provident by forecasting possible future classes.These properties avoid squeezing former classes and reduce the effects of future updating shocks.
  • FACT: FACT assigns virtual prototypes to reserve embedding space, uses virtual instances to forecast possible classes, and uses the prototypes as basis vectors during incremental inference.The method prepares the base model for future classes while supporting stronger incremental classification.

2. Related Work

The paper situates FSCIL among few-shot learning and class-incremental learning, whose methods address scarce examples and sequential new classes without forgetting old ones.

  • Few-Shot Learning: Few-shot learning targets unseen classes with insufficient training instances using optimization-based or metric-based approaches.Metric-based methods commonly use a pretrained backbone and distances between support and query instances.
  • Class-Incremental Learning: Class-incremental learning targets sequences of new classes without forgetting old ones through parameter protection, distillation, or rehearsal.The related-work discussion also notes that pre-allocating future classifiers can require extra memory unsuitable for FSCIL.

3. From Old Classes to New Classes

FSCIL begins with a data-rich base session and proceeds through disjoint few-shot incremental sessions, requiring new-class learning while preserving old-class performance. Standard and backward-compatible baselines address this sequential setting but do not prepare the embedding space prospectively.

  • FSCIL Setting: The base session provides sufficient instances for classes Y0, while later sessions provide only limited N-way K-shot datasets with disjoint label spaces.The model accesses only the current incremental dataset while learning each task.
  • Model Formulation: The model decomposes into an embedding module φ and linear classifier W, with each class represented by a classifier vector wk.The classifier is written as f(x) = W⊤φ(x).
  • FSCIL Setting: At each incremental session, the model must learn new classes and maintain performance over all previously seen classes.The objective is expressed as minimizing empirical risk over the testing datasets associated with the sequence.
  • Backward-Compatible Baselines: Knowledge distillation aligns old and current model outputs, while ProtoNet fixes the embedding and uses averaged class embeddings as prototypes.These strategies aim to preserve backward compatibility during incremental updates.
  • Backward-Compatible Baselines: Retrospective backward-compatibility methods maintain old-class behavior or freeze embeddings but do not explicitly prepare for future classes.The paper motivates shifting preparation toward the base session and future extensions.

4. Forward Compatible Training for FSCIL

FACT prepares FSCIL models prospectively by reserving embedding space for future classes and forecasting possible incoming classes. Virtual prototypes and mixed virtual instances shape a bimodal embedding structure that supports incremental inference while preserving known-class compactness.

  • Allocating Virtual Prototypes: FACT assigns each instance to its ground-truth class and an extra virtual class, optimizing a bimodal target that reserves space for future classes.The virtual label is selected from the highest-logit virtual prototype after masking the ground-truth class.
  • Allocating Virtual Prototypes: Multiple virtual prototypes act as reserved class spaces, pushing known-class embeddings toward compact clusters and leaving room for incoming classes.The default number of virtual classes is set to the number of new classes, V = NB.
  • Forecasting Virtual Instances: FACT forecasts possible future classes by applying manifold mixup to instances from different classes and treating the interpolated embedding as a virtual instance.The embedding is decomposed as φ(x) = g(h(x)), then hidden representations are interpolated using λ sampled from a Beta distribution.
  • Forecasting Virtual Instances: The forecasting loss symmetrically assigns each mixed instance toward both a virtual prototype and its nearest known class, preventing known classes from being over-squeezed.The virtual-class term reserves space, while the known-class term trades off between virtual and known classes.
  • Incremental Inference with Virtual Prototypes: During inference, FACT expands the classifier with prototypes of newly arriving classes and incorporates the influence of informative virtual prototypes into prediction.The learned virtual prototypes provide bases for a stronger incremental classifier from the more informative embedding distribution.

5. Experiment

FACT is evaluated on benchmark and large-scale FSCIL datasets, with experiments covering comparisons, ablations, visualizations, and sensitivity analyses. It consistently outperforms strong baselines while reserving embedding space for future classes and resisting forgetting.

  • Benchmark Comparison: FACT surpasses CEC on every reported Top-1 and Top-5 metric for ImageNet100 and ImageNet1000.The result extends FACT’s performance advantage from small-scale benchmarks to large-scale FSCIL tasks.
  • Ablation Study: Ablations show that virtual prototypes, forecasting, and the remaining FACT components each improve FSCIL performance and forward compatibility.Without future-class modeling, L1 overspreads known-class embeddings; L2 reserves space with virtual prototypes, while subsequent components further improve the model.
  • Visualization of Incremental Sessions: Decision-boundary visualizations show compact known-class embeddings and reserved regions that let new classes avoid squeezing old-class representations.Virtual prototypes are assigned to one or more known classes and cover the known classes in preparation for possible updates.
  • Additional Analyses: FACT performs better on new classes, favors small γ and large V, and improves final accuracy as incremental shots increase.The suggested defaults are γ = 0.01 and V = NB, while the shot analysis varies K across {1, 5, 10, 15, 20, 50}.

6. Conclusion

The paper presents FACT as a forward-compatible FSCIL approach that reserves and forecasts embedding space for future classes. It reports state-of-the-art performance while noting that sufficient new-class data changes the setting toward CIL.

  • Conclusion: FACT uses virtual prototypes to reserve embedding space and forecast possible future classes, making the model growable and provident.Virtual prototypes also act as embedding-space bases that facilitate FSCIL inference.
  • Conclusion: FACT incorporates new knowledge while resisting forgetting of old classes and achieves state-of-the-art FSCIL performance.The conclusion describes this as efficient incorporation of new knowledge with forward compatibility.
  • Limitations: The method assumes few-shot new-class instances; with sufficient instances, the problem becomes CIL and forward and backward compatibility should be considered together.This is the paper’s stated limitation and scope boundary.

I. Gradient Analysis

The supplementary analysis formalizes FACT’s model output, embedding decomposition, virtual-prototype classifier, and combined loss. It sets up gradient analyses for the four loss terms.

  • Gradient Setup: The analysis covers gradients with respect to embeddings, classifier weights, and the non-identity embedding function case.It extends the main-paper discussion of L1, L2, L3, and L4.
  • Model Formulation: FACT represents the output as a unified classifier over learned and virtual prototypes applied to an embedding φ(x) = g(h(x)).The virtual-prototype collection is denoted P_v, and the final loss combines L_v and L_f.
  • Gradient Setup: The gradient analysis treats the classifier [W, P_v] as a unified classifier and uses cross-entropy discrepancy for subsequent derivations.The supplementary analysis explicitly states this simplifying assumption for discussing gradients.

I.1. Supplementary Analysis for the Main Paper

The supplementary derivations explain how FACT’s losses shape embeddings and prototypes. L2 and L3 reserve space for future classes, while symmetric L4 regularizes their combined squeezing effect.

  • L1 Analysis: L1 pulls known-class embeddings toward their target prototypes and away from others, providing ordinary classification discriminability.Its prototype gradients similarly attract target prototypes and repel non-target prototypes.
  • L2 Analysis: L2 pushes embeddings toward assigned virtual prototypes and away from other prototypes, explicitly reserving space without weakening known-class classification.The ground-truth probability is masked during L2 optimization, so the push does not affect the ground-truth class.
  • L3 Analysis: L3 applies analogous forces to manifold-mixup virtual instances, making the model provident by reserving space through generated examples.The mixed embedding and its component embeddings are directed toward the pseudo-labeled prototype and away from other prototypes.
  • L4 Analysis: L4 produces a symmetric loss resembling L1 and trades off space squeezing against over-squeezing caused by L2 and L3.The main-paper ablations validate this regularization effect.

I.2. When g(·) is not Identity

The analysis extends FACT beyond the identity case by considering a general non-identity function g(·), beginning with linear and nonlinear settings.

  • The section studies FACT when g(·) is not the identity function.
  • It first analyzes the linear case before treating the nonlinear case.
  • Because the gradients of L3 and L4 are similar, only L3’s gradients are presented as an example.

I.2.1 g(·) is Linear

For linear classifiers, optimizing L3 reserves embedding space by moving mixed-instance embeddings toward virtual-class prototypes and away from others, with the effect propagating through the network. The analysis further states that forward compatibility is maintained for nonlinear classifiers and that virtual prototypes strengthen inference classification.

  • I.2.1 g(·) is Linear: For a linear layer g(·), L3 moves the mixed-instance embedding toward the virtual class prototype and away from other prototypes.The mixed embedding is z = V⊤b, where b mixes two middle-layer representations.
  • I.2.1 g(·) is Linear: The reserving effect propagates from the final embedding into the middle layer and from shallow to deep representations.This holistic propagation is presented as maintaining forward compatibility throughout the network.
  • I.2.1 g(·) is Linear: Even with a linear classifier, forward compatibility remains maintained with L3 because the gradient adds an extra term in the gradient direction.The conclusion is reported as consistent with Eq. 12.
  • I.2.1 g(·) is Linear: For nonlinear g(·), the same conclusion holds when the relevant Jacobian-transformed prototype directions are similar, such as having high cosine similarity.The nonlinear analysis identifies the Jacobian of g(b) with respect to b as part of the condition.
  • I.2.1 g(·) is Linear: FACT inference can incorporate virtual prototypes as proxies that encode classification ability across known classes and build a stronger classifier than ProtoNet under the same loss.The ablation comparison reports better inference performance for FACT than ProtoNet.
  • I.2.1 g(·) is Linear: FACT reports higher top-1 accuracy and lower performance decay than other state-of-the-art methods, indicating less forgetting across benchmark datasets.On CIFAR100, FACT exceeds the runner-up by 2.96% in last accuracy and 1.43% in performance decay.
Loading 2203.06953v1…