PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learning

Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, Dacheng Tao

arXiv:2302.03004v1cs.CVcs.LG

TL;DR

FSCIL must learn novel classes from few samples while retaining old-class performance, but backbone finetuning or prototype adjustment can misalign features and classifiers. The paper fixes a simplex-ETF classifier for the entire label space and trains features toward corresponding prototypes, with theory and experiments supporting preserved alignment and strong benchmark performance.

  • Problem

    FSCIL provides only a few samples for each novel class, while finetuning features or adjusting old prototypes can create feature-classifier misalignment associated with catastrophic forgetting.

  • Method

    The method pre-assigns fixed simplex-ETF classifier prototypes for all base and incremental classes and uses a novel loss to drive features toward their corresponding prototypes.

  • Results

    The method surpasses state-of-the-art performance across three FSCIL datasets, with more than 3.5% average accuracy improvement over a strong baseline on miniImageNet and CIFAR-100.

  • Takeaways & Limitations

    A fixed neural-collapse-inspired target can preserve feature-classifier alignment during incremental training and relieve forgetting within the evaluated FSCIL setting.

  • Takeaways & Limitations

    The theoretical analysis uses a simplified model that drops the backbone and treats last-layer features and classifier prototypes as independent optimization variables.

Abstract

from arXiv · show

Few-shot class-incremental learning (FSCIL) has been a challenging problem as only a few training samples are accessible for each novel class in the new sessions. Finetuning the backbone or adjusting the classifier prototypes trained in the prior sessions would inevitably cause a misalignment between the feature and classifier of old classes, which explains the well-known catastrophic forgetting problem. In this paper, we deal with this misalignment dilemma in FSCIL inspired by the recently discovered phenomenon named neural collapse, which reveals that the last-layer features of the same class will collapse into a vertex, and the vertices of all classes are aligned with the classifier prototypes, which are formed as a simplex equiangular tight frame (ETF). It corresponds to an optimal geometric structure for classification due to the maximized Fisher Discriminant Ratio. We propose a neural collapse inspired framework for FSCIL. A group of classifier prototypes are pre-assigned as a simplex ETF for the whole label space, including the base session and all the incremental sessions. During training, the classifier prototypes are not learnable, and we adopt a novel loss function that drives the features into their corresponding prototypes. Theoretical analysis shows that our method holds the neural collapse optimality and does not break the feature-classifier alignment in an incremental fashion. Experiments on the miniImageNet, CUB-200, and CIFAR-100 datasets demonstrate that our proposed framework outperforms the state-of-the-art performances. Code address: https://github.com/NeuralCollapseApplications/FSCIL

1 INTRODUCTION

FSCIL must learn new classes from very few samples while retaining performance on previously learned classes. The paper frames catastrophic forgetting as a feature-classifier misalignment problem and proposes fixing a neural-collapse-inspired alignment across sessions.

  • Motivation: FSCIL combines continual learning over new label spaces with only a few training samples per novel class.Unlike standard few-shot learning and incremental learning, evaluation includes all encountered classes.
  • Motivation: Finetuning the backbone or incrementally adjusting classifier prototypes can misalign old-class features and classifiers, harming old-class discrimination.Prior methods adjust prototypes to separate old and novel classes, but this adjustment can conflict with fixed features.
  • Motivation: Neural collapse aligns same-class features with classifier prototypes arranged as a simplex ETF, providing an optimal geometric structure for classification.The structure is associated with minimized within-class variance and maximized between-class variance.
  • Proposed approach: The proposed framework pre-assigns fixed classifier prototypes for the whole label space and trains features toward the same neural-collapse optimality in every session.This fixed target is intended to avoid optimization conflict among sessions.
  • Results: The paper reports theoretical and empirical support for preserving alignment incrementally and surpassing state-of-the-art FSCIL performance on miniImageNet, CIFAR-100, and CUB-200.The contribution summary specifically reports average accuracy gains exceeding 3.5% over a strong baseline on miniImageNet and CIFAR-100.

2 RELATED WORK

The related work situates FSCIL at the intersection of class-incremental and few-shot learning, where adapting to novel classes must be balanced against forgetting. It also connects neural collapse theory to optimal feature-classifier geometry under balanced training.

  • Few-shot class-incremental learning: FSCIL extends class-incremental learning with only a few novel-class samples per incremental session, increasing the risk of overfitting to novel classes.Both FSCIL and CIL require balancing adaptation to novel classes with retention of old-class knowledge.
  • Neural collapse: Neural collapse describes an elegant last-layer feature-classifier geometry that prior work has analyzed as a global optimum under balanced cross-entropy or mean-squared-error training.The cited theory focuses on simplified last-layer optimization models.

3 BACKGROUND

FSCIL trains across disjoint class sessions with abundant base-session data but few labeled examples in each incremental session, while testing on all encountered classes. Neural collapse provides the geometric background: same-class features collapse, class means form a simplex ETF, and classifier weights align with them.

  • FSCIL setting: FSCIL uses a base dataset with many classes and sufficient examples, followed by disjoint incremental sessions with p-way q-shot data.Previous-session training sets are unavailable during incremental training, and evaluation covers all classes encountered so far.
  • FSCIL setting: The setting creates severe data scarcity and imbalance while requiring adaptation to novel classes without losing old-class ability.The model must remain effective on both novel and previously encountered classes.
  • Neural collapse: Neural collapse is defined at terminal training after zero training error on balanced data and describes a geometric structure involving last-layer features and classifiers.The definition introduces the subsequent neural-collapse properties.
  • Simplex ETF: A simplex ETF consists of K vectors in R^d with equal norm and equal pairwise inner products, allowing a highly symmetric class geometry.The supplied definition states that the matrix may include a rotation and uses an identity matrix and an all-ones vector.
  • Neural-collapse properties: Neural collapse includes within-class feature collapse and simplex-ETF arrangement of centered class means.The within-class covariance collapses, while centered class means satisfy the simplex ETF relation.
  • Neural-collapse properties: Centered within-class means align with their corresponding classifier weights, so prediction can be reduced to choosing the nearest class center.The equivalence is expressed through inner-product maximization and Euclidean-distance minimization.
  • Motivation: This alignment is considered optimal because it maximizes the Fisher Discriminant Ratio, but imbalanced or incremental training can break the neural-collapse structure.The paper uses this breakdown to motivate preserving the geometry explicitly in FSCIL.

4 METHOD

NC-FSCIL fixes an ETF classifier over the entire label space and trains features toward their corresponding prototypes, preserving the neural-collapse geometry during incremental learning. Theoretical analysis supports this alignment under imbalanced class and sample counts.

  • 4.1 ETF CLASSIFIER: The method pre-assigns classifier prototypes for the whole label space as a simplex ETF and keeps them fixed throughout incremental training.The ETF provides maximally separated prototypes and a fixed target for aligning output features.
  • 4.2 DOT-REGRESSION LOSS: The dot-regression loss pulls each normalized feature toward its same-class prototype, while omitting the potentially inaccurate push term used to separate other classes.Its feature gradient is −(1 − cos ∠(ˆµi, ˆwyi)) ˆwyi, directing the feature toward the pre-assigned target.
  • 4 METHOD: NC-FSCIL uses a backbone and projection layer to produce normalized output features that are trained against the fixed ETF prototypes.The normalized feature is computed as ˆµi = µi/∥µi∥, with µi = g(hi, θg) and hi = f(xi, θf).
  • 4 METHOD: During the base session, both backbone and projection layer are trained; in incremental sessions, the backbone is fixed and only the projection layer is finetuned.A memory of mean intermediate features for old classes is used alongside novel-session data during incremental optimization.
  • 4.4 THEORETICAL SUPPORTS: Theoretical analysis shows that the incremental global minimizer satisfies neural collapse, with class features aligned to simplex-ETF prototypes despite imbalanced classes and sample counts.The result applies when the loss is cross entropy or dot regression and establishes equal separation by prototypes across the label space.

5 EXPERIMENTS

Experiments evaluate the method on FSCIL benchmarks, compare it with prior studies, test ETF and DR components, and examine feature-classifier alignment. The method improves benchmark performance and maintains more stable alignment during incremental training.

  • 5 EXPERIMENTS: The experiments use miniImageNet, CIFAR-100, and CUB-200, with results reported through session-wise benchmark tables and ablation studies.The implementation details are provided in Appendix B.
  • 5.2 PERFORMANCE ON BENCHMARKS: The method achieves the best performance across all sessions on miniImageNet and CIFAR-100, while obtaining the best average accuracy on CUB-200.Against ALICE, it improves last-session accuracy by 2.61% on miniImageNet and 2.01% on CIFAR-100, with average accuracy improvements exceeding 3.5% on both datasets.
  • 5.3 ABLATION STUDIES: Fixed ETF prototypes substantially improve final-session accuracy and mitigate performance drops, while DR loss provides a further moderate improvement.The ablation compares Learnable+CE, ETF+CE, and ETF+DR across FINAL, AVERAGE, and PD metrics.
  • 5.4 FEATURE-CLASSIFIER STRUCTURE: The method keeps different-class feature-prototype cosine similarities mostly negative and near zero, unlike the baseline’s increasing similarities across sessions.This indicates a more stable feature-classifier alignment and preserves the margin between different classes during incremental training.

6 CONCLUSION

The paper fixes a neural-collapse-inspired geometric classifier structure and trains features toward it throughout incremental learning. Theoretical and empirical results support preserved alignment and improved FSCIL performance across three datasets.

  • 6 CONCLUSION: The method pre-assigns an optimal feature-classifier alignment as a fixed target throughout incremental learning to avoid optimization conflicts among sessions.A novel loss drives features toward the fixed prototypes without a regularizer.
  • 6 CONCLUSION: Theoretical and empirical analyses indicate that the method holds neural collapse alignment incrementally and relieves forgetting.

STATEMENTS

The paper reports no ethics concerns and provides reproducibility materials including a theorem proof, public datasets, implementation details, and released source code.

  • Ethics Statement: The study reports no involvement of human subjects, public health, privacy, fairness, or security issues and states compliance with the ICLR Code of Ethics.
  • Reproducibility Statement: Theorem 1 is proved in Appendix A, datasets are public and cited, implementation details are in Appendix B, and source code is released.

A APPENDIX: PROOF OF THEOREM 1

The appendix proves that, with a fixed ETF classifier, the optimized features align with their corresponding prototypes and retain simplex-ETF relationships across sessions.

  • Proof setup: The theoretical analysis uses a simplified model with last-layer features and classifier prototypes as independent optimization variables.The proof studies an incremental problem over T sessions with a fixed ETF classifier.
  • ETF structure: The ETF classifier contains K prototypes whose pairwise inner products are −1/(K−1), with zero sum across all prototypes.This is the simplex-ETF geometry used throughout the proof.
  • Cross-entropy proof: For cross-entropy loss, convexity with respect to constrained features permits global-optimality analysis through the KKT conditions.The objective is separable across sessions when the ETF classifier is fixed.
  • Cross-entropy proof: At the optimum, each feature has unit norm and is aligned with its corresponding prototype, yielding cos ∠(m̂_k,i, ŵ_k)=1.The proof explicitly concludes that feature vectors align with their class prototypes.
  • Cross-entropy proof: Each optimized feature preserves the ETF relation to every other prototype, with inner product −1/(K−1) for all mismatched classes.The resulting feature-prototype relations match the simplex-ETF geometry.

B APPENDIX: IMPLEMENTATION DETAILS

The experiments use three FSCIL benchmarks with dataset-specific image settings and backbones, plus a projection layer for the proposed method.

  • Datasets: Experiments use miniImageNet, CIFAR-100, and CUB-200 as FSCIL benchmark datasets.miniImageNet and CIFAR-100 each contain 100 classes; CUB-200 is a fine-grained image-classification dataset.
  • Datasets: miniImageNet images are 84 × 84, while CIFAR-100 images are 32 × 32; both datasets have 500 training and 100 testing images per class.The passage describes these dataset statistics explicitly.
  • Architectures: ResNet-12 is used for miniImageNet and CIFAR-100, while ImageNet-pretrained ResNet-18 is used for CUB-200.The backbone choices follow prior FSCIL studies.
  • Architectures: A two-layer MLP block serves as the projection layer.The projection-layer design follows the practice reported in Peng et al. (2022).
  • Training details: Training uses standard random resizing, random flipping, and color jittering, with batch sizes of 512 in the base session and 64 in incremental sessions.Incremental batches combine new-session data with intermediate features retained in memory.

C APPENDIX: MORE RESULTS

Additional experiments show that the method maintains stronger feature-classifier alignment and lower within-to-between-class covariance ratios than the baseline, including for base-session classes.

  • CUB-200 results: On CUB-200, the method achieves better last-session accuracy than most baselines and the best average accuracy among all methods.It does not surpass ALICE in last-session accuracy on CUB-200.
  • Feature-classifier alignment: The method consistently produces higher same-class feature-prototype cosine similarity than the baseline across session-level train and test evaluations.Higher similarity indicates closer alignment between feature centers and corresponding prototypes.
  • Feature-classifier alignment: Across all encountered classes, the method’s alignment metric remains largely stable after the fourth session on the train set, while the baseline keeps decreasing.For base-session classes, the method remains stable after the initial decline over the first three to four sessions on both train and test sets.
  • Overall interpretation: The alignment and covariance results support maintaining feature-classifier structure for base-session classes during incremental training.The appendix connects stable base-class geometry with preservation of performance on old classes.
  • Class covariance: The method has a consistently lower within-class-to-between-class covariance trace ratio than the baseline.For base-session classes, its ratio increases more mildly, supporting better maintenance of old-class performance.
Loading 2302.03004v1…