PaperlayerView exact revision on arXiv

Source-linked AI summary

Constrained Few-shot Class-incremental Learning

Michael Hersche, Geethan Karunaratne, Giovanni Cherubini, Luca Benini, Abu Sebastian, Abbas Rahimi

arXiv:2203.16588v1cs.CVcs.LG

TL;DR

Few-shot class-incremental learning must absorb novel classes from limited fresh data without forgetting old classes while keeping computation constant and memory growth linear. C-FSCIL uses a frozen meta-learned extractor, fixed-size trainable layer, and growing prototype memory with hyperdimensional quasi-orthogonal representations. It outperforms baselines across CIFAR100, miniImageNet, and Omniglot, scaling to 1623 classes with mode-dependent accuracy drops below 2.6%.

  • Problem

    Few-shot class-incremental learning must learn novel classes from online data with few samples while constraining computational cost and memory growth and avoiding catastrophic forgetting.

  • Method

    C-FSCIL combines a frozen meta-learned feature extractor, trainable fixed-size fully connected layer, dynamically growing explicit memory, and hyperdimensional quasi-orthogonal prototypes updated through three modes.

  • Results

    C-FSCIL outperforms FSCIL baselines across CIFAR100, miniImageNet, and Omniglot, reaching 1623 Omniglot classes with less than 2.6%, 1.4%, and 1.6% accuracy drops for Modes 1, 2, and 3.

  • Takeaways & Limitations

    One-pass averaged prototypes already outperform other methods, while Modes 2 and 3 achieve up to 4% higher accuracy with at most 50 iterations.

  • Takeaways & Limitations

    Each incremental session provides access only to its current c-way k-shot training set, while evaluation includes classes from all previous sessions and the current one.

Abstract

from arXiv · show

Continually learning new classes from fresh data without forgetting previous knowledge of old classes is a very challenging research problem. Moreover, it is imperative that such learning must respect certain memory and computational constraints such as (i) training samples are limited to only a few per class, (ii) the computational cost of learning a novel class remains constant, and (iii) the memory footprint of the model grows at most linearly with the number of classes observed. To meet the above constraints, we propose C-FSCIL, which is architecturally composed of a frozen meta-learned feature extractor, a trainable fixed-size fully connected layer, and a rewritable dynamically growing memory that stores as many vectors as the number of encountered classes. C-FSCIL provides three update modes that offer a trade-off between accuracy and compute-memory cost of learning novel classes. C-FSCIL exploits hyperdimensional embedding that allows to continually express many more classes than the fixed dimensions in the vector space, with minimal interference. The quality of class vector representations is further improved by aligning them quasi-orthogonally to each other by means of novel loss functions. Experiments on the CIFAR100, miniImageNet, and Omniglot datasets show that C-FSCIL outperforms the baselines with remarkable accuracy and compression. It also scales up to the largest problem size ever tried in this few-shot setting by learning 423 novel classes on top of 1200 base classes with less than 1.6% accuracy drop. Our code is available at https://github.com/IBM/constrained-FSCIL.

1. Introduction

C-FSCIL addresses few-shot class-incremental learning under constrained samples, computation, and memory by combining hyperdimensional representations with a modular architecture and quasi-orthogonal class prototypes. Its update modes trade accuracy against resource costs while achieving strong accuracy, compression, and scalability.

  • Few-shot class-incremental learning must add novel classes from online data streams while constraining sample size, computation, and memory.
  • Catastrophic forgetting occurs when sequential learning overwrites weights representing old classes, reducing classification accuracy.
  • Hyperdimensional vectors are likely to be quasi-orthogonal, minimizing interference while providing capacity for many incremental classes.Their capacity grows exponentially with dimensionality, while semantic information can still be encoded.
  • C-FSCIL combines a frozen feature extractor, trainable fixed-size fully connected layer, and dynamically growing explicit memory storing one vector per encountered class.The embedding dimensionality remains fixed independently of the number of past and future classes.
  • Three update modes trade accuracy against compute-memory cost, ranging from one-pass averaged prototypes without gradients to quasi-orthogonal prototype alignment with bounded retraining.Mode 3 uses novel loss functions to align prototypes while keeping them near their original averaged representations.
  • 423 novel classes added to 1200 base classes with less than 2.6%, 1.4%, and 1.6% accuracy drops for Modes 1, 2, and 3, respectively, while 2× prototype compression caused 1.7%–3.5% drops.Experiments covered CIFAR100, miniImageNet, and Omniglot, where C-FSCIL outperformed baselines.

2. Related Work

Prior approaches address continual few-shot learning through meta-learned embedding networks, rehearsal memories, metric or bias corrections, and specialized adaptation modules. C-FSCIL instead stores compressed class knowledge and uses hyperdimensional quasi-orthogonal prototypes to reduce interference while supporting more classes than the embedding dimension.

  • Memory-augmented neural networks separate information processing from memory storage, enabling few-shot class prototypes to be stored and retrieved explicitly.Their embedding networks are meta-learned on base classes before representing novel categories.
  • Rehearsal methods retain old exemplars and replay them with current data during incremental learning, keeping prior training information available to the classifier.
  • C-FSCIL avoids rehearsal and pattern replay by retaining compressed past knowledge in explicit or globally averaged activation memory.In Modes 2–3, globally averaged activations can rematerialize the explicit memory on demand.
  • Class-incremental methods address imbalance and interference using cosine distance, bias correction, weight alignment, regularization, or gradient masking.These methods target norm or bias imbalance and changes to past-class weights.
  • Hyperdimensional quasi-orthogonal class vectors reduce interference and support more classes than fixed Hadamard or identity-layer alternatives.C-FSCIL stores prototypes in explicit memory, compares them with cosine similarity, and selectively updates them.
  • Few-shot class-incremental learning methods include neural gas, graph attention, semantic word embeddings, and vector quantization.

3. Notations and Preliminaries

FSCIL presents sequential, mutually exclusive training sessions: a larger base session followed by few-shot novel sessions without access to earlier training data. Evaluation accumulates classes from previous and current sessions.

  • FSCIL provides training sets sequentially, with mutually exclusive class sets across sessions.
  • The base session supplies more training examples and classes to learn a representation distinguishing its classes.
  • Each later session is c-way k-shot, containing c classes and k training samples per class.
  • During session s, only the current training set is available; earlier session training sets are no longer accessible.
  • The evaluation set for a session contains samples from all previous sessions and the current session.

4. Proposed Method: C-FSCIL

C-FSCIL combines a feature extractor, trainable fully connected layer, and explicit prototype memory in a hyperdimensional embedding architecture. Its three update modes progressively trade computation and memory for improved class separation while supporting continual few-shot learning.

  • Architecture: C-FSCIL uses a feature extractor, a fixed-dimensional trainable fully connected layer, and an explicit memory for class prototypes.The feature extractor can be a five-layer CNN or ResNet-12, while the fully connected output dimension satisfies d ≤ 512.
  • Architecture: Prototype-based inference compares a query embedding with stored prototypes using tanh activation and cosine similarity.The resulting attention vector reads the one-hot label memory.
  • Meta-learning: Meta-learning trains the embedding network and explicit memory to assign different classes nearly quasi-orthogonal vectors in hyperdimensional space.The embedding space is intended to provide dissimilar vectors for newly encountered classes.
  • Mode 1: Mode 1 learns novel classes online without gradient-based parameter updates, storing class prototypes in an explicit memory that grows with encountered classes.The architecture figure distinguishes component updates across meta-learning, base, and novel sessions; all modes require prior meta-learning.
  • Modes 2 and 3: Modes 2 and 3 improve inter-class separation by adjusting prototypes and retraining the fully connected layer using globally averaged activations.Mode 2 bipolarizes prototypes with an element-wise sign operation before alignment-based retraining; the globally averaged activation memory supports later modes.
  • Mode 3: Mode 3 minimizes prototype cross-correlation to obtain quasi-orthogonal vectors even when the number of classes exceeds the embedding dimension.The proposed loss penalizes large absolute cross-correlations, while exact orthogonality is not possible when d < | ˜C(s)|.

5. Experiments

Experiments on miniImageNet, CIFAR100, and Omniglot evaluate C-FSCIL across few-shot incremental settings and update modes. C-FSCIL achieves strong accuracy, scalability, and efficiency, with performance varying by the number of classes and update sophistication.

  • Experimental setup: C-FSCIL is evaluated on miniImageNet and CIFAR100 with 60 base classes followed by eight 5-way 5-shot novel sessions.
  • Experimental setup: C-FSCIL is evaluated on Omniglot with 1200 base classes and nine 47-way 5-shot novel sessions, reaching 1623 total classes.
  • Comparative results: C-FSCIL sets the state of the art on miniImageNet and CIFAR100, while one-pass Mode 1 outperforms all compared methods without retraining or auxiliary GAA memory.
  • Comparative results: Prototype quasi-orthogonalization consistently improves accuracy; Mode 2 is more effective for fewer classes, whereas Mode 3 performs best for larger class counts.
  • Comparative results: In Omniglot, Mode 1 declines from 84.16% in the base session to 81.56% in the last session, while Mode 3 reaches 87.21% and 85.70%, respectively.
  • Comparative results: All C-FSCIL modes outperform ANML in an alternative Omniglot setting with 600 novel classes, which excludes evaluation on base classes.
  • Ablation study: Mode 3 reduces the embedding dimension below the total class count with marginal accuracy degradation while retaining performance above all baselines.

6. Conclusion and Outlook

C-FSCIL supports rapid few-shot class-incremental learning under constrained updates and memory growth. Its prototype-based representations also support robustness and reduced-precision implementations.

  • C-FSCIL learns novel classes either in one pass without gradient updates or with a small constant number of retraining iterations.Modes 2 and 3 use at most 50 iterations.
  • The C-FSCIL memory grows at most linearly with the number of encountered classes.
  • Simple averaged prototypes in Mode 1 outperform other methods on CIFAR100, miniImageNet, and Omniglot.
  • Modes 2 and 3 improve accuracy by up to 4% when the maximum number of classes is encountered.
  • Large inter-class separation provides robustness against adversarial perturbations without adversarial training, while bipolarized prototypes support reduced-precision implementations.

A.1. Datasets

The experiments use miniImageNet, CIFAR100, and Omniglot under few-shot class-incremental protocols. The datasets differ substantially in image format, class count, and base-to-novel class allocation.

  • miniImageNet: miniImageNet contains 100 classes of 84×84 RGB images and uses 60 base classes plus eight 5-way 5-shot novel sessions.
  • CIFAR100: CIFAR100 contains 100 classes of 32×32 images and uses 60 base classes plus eight 5-way 5-shot novel sessions.
  • Omniglot: Omniglot contains 1623 classes with 20 binary 105×105 images per class, using 1200 base classes and 423 novel classes.
  • Omniglot protocol: For Omniglot, six base samples are held out, five samples from each 47-way novel batch provide support, and nine subsequent sessions yield 1623 final classes.

A.2.1 Reducing Dimension

The dimension ablation shows that preferred embedding size depends on the update mode and number of classes. Mode 3 can retain strong performance below the class count, whereas Omniglot favors its largest tested dimension.

  • Mode 3 achieves its best last-session accuracy with d = 128 on miniImageNet at 51.46% and CIFAR100 at 50.74%.
  • On Omniglot, d = 512 gives the highest accuracy for all modes because the tested maximum dimension remains below the number of classes.
  • C-FSCIL maintains superior accuracy to state-of-the-art methods even when d < |C̃(S)|.
  • With d = 64, C-FSCIL outperforms the listed baselines on miniImageNet in Modes 1 and 3 and on CIFAR100 across the reported modes.
  • With d = 128 in any mode, C-FSCIL outperforms ProtoNet and CEC on Omniglot.

A.2.2 Other Attention Functions

The appendix compares softabs and softmax attention, showing how their sharpening and nudging objectives shape prototype correlations and incremental accuracy. Softabs promotes quasi-orthogonality, while softmax promotes anti-correlating prototypes.

  • Softabs attention: Softabs attention sharpens cosine similarities to promote quasi-orthogonal prototypes during meta-learning.Its sharpening function is maximized at cosine similarities of 1 or −1 and minimized at 0.
  • Softmax attention: Softmax attention uses exponential weighting with inverse temperature τ = 10 and categorical cross-entropy to seek anti-correlating prototypes.The comparison includes a grid-search over the inverse softmax temperature.
  • Accuracy comparison: On miniImageNet, softmax starts 0.1% above softabs in the base session but decays faster as novel classes are added.The appendix compares classification accuracy across miniImageNet, CIFAR100, and Omniglot.
  • Prototype correlations: In Mode 1, softabs yields near-zero cross-correlations on base classes, whereas softmax produces negative cross-correlations.The comparison is illustrated in Fig. A1 for miniImageNet.
  • Prototype nudging: Mode 3 exponential nudging produces anti-correlations between novel and base prototypes, while double-exponential nudging produces quasi-orthogonal prototypes.Both approaches reduce cross-talk on novel classes, but some base-session discriminability is sacrificed.

A.2.3 Smaller Feature Extractor

This section evaluates smaller feature extractors and memory compression for C-FSCIL. The smaller ResNet-12 maintains near-original accuracy with fewer trainable parameters, while compression preserves Mode 1 accuracy better than Mode 3 accuracy.

  • Smaller feature extractor: ResNet-12 (small) has 8.0 M parameters and 1.56× fewer trainable parameters than ResNet-18.Its block dimensions are [64, 128, 256, df = 512].
  • Smaller feature extractor: Accuracy drops by less than 1% with ResNet-12 (small) compared with the original ResNet-12 on miniImageNet and CIFAR100.The comparison is reported in Tables A7 and A8.
  • Smaller feature extractor: C-FSCIL with ResNet-12 (small) outperforms all miniImageNet baselines in Modes 1–3 while using fewer trainable parameters.On CIFAR100, it outperforms the majority of listed baselines and exceeds CEC in Modes 2 and 3.
  • Memory compression: A 2× memory compression uses randomly keyed holographic reduced representations to superimpose two key-prototype pairs.The keys can be regenerated from stored 32-bit seeds, while circular correlation retrieves prototypes with dimension-dependent noise.
  • Memory compression: Compressed EM in Mode 1 loses 1.7%–3.5% across sessions, whereas compressed GAA memory in Mode 3 loses 4.7%–8.5%.The larger Mode 3 loss may reflect the GAA memory’s lack of quasi-orthogonal representation.
  • Continually Evolved Classifiers: C-FSCIL Mode 3 exceeds CEC by 8.30% in session 1 and 9.59% in session 10.CEC achieved its highest reported accuracy at embedding dimension d = 64.
  • Alternative FSCIL setting on Omniglot: In the alternative Omniglot setting, Mode 3 has a 10.1% accuracy drop compared with ANML’s 31.1% drop as classes increase from 10 to 600.C-FSCIL consistently performs better than ANML in this setting.
Loading 2203.16588v1…