PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Role of Neural Collapse in Transfer Learning

Tomer Galanti, András György, Marcus Hutter

arXiv:2112.15121v2cs.LG

TL;DR

The paper addresses limited theoretical understanding of why foundation-model features transfer between tasks. It analyzes this through neural collapse and reports that the property generalizes to new samples and unseen classes, with few samples sufficient to train a linear classifier on the learned features.

  • Problem

    Why transfer learning with foundation models works across tasks is relatively little understood, despite its use when target-task data are scarce.

    Source: §1 · pp. 1–2

  • Method

    The paper analyzes transfer learning through neural collapse, studying source-trained features and few-shot target classification both theoretically and experimentally.

    Source: §1 · p. 2, §5.1 · p. 8

  • Results

    Neural collapse generalizes to new samples and, when new classes share the training classes' distribution, to unseen classes where few samples can suffice for training a linear classifier.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The findings provide an explanation for foundation models' empirical success in transfer learning, including few-shot classification.

    Source: §1 · p. 2

    The reported comparison with Distill-simple is likely affected by the ridge-regression target classifier, which is inferior to its logistic-regression solution.

    Source: §5.2 · p. 8

Abstract

from arXiv · show

We study the ability of foundation models to learn representations for classification that are transferable to new, unseen classes. Recent results in the literature show that representations learned by a single classifier over many classes are competitive on few-shot learning problems with representations learned by special-purpose algorithms designed for such problems. In this paper we provide an explanation for this behavior based on the recently observed phenomenon that the features learned by overparameterized classification networks show an interesting clustering property, called neural collapse. We demonstrate both theoretically and empirically that neural collapse generalizes to new samples from the training classes, and -- more importantly -- to new classes as well, allowing foundation models to provide feature maps that work well in transfer learning and, specifically, in the few-shot setting.

1 INTRODUCTION

Transfer learning uses representations pretrained on broad source tasks to address target tasks with scarce data, but the theoretical basis for its success remains limited. The paper connects neural collapse to transferability and studies adaptation from one source classification task to unseen tasks using few samples.

1 INTRODUCTION

  • Transfer learning reuses features from a large source classification task to train a smaller target classifier when target data are scarce.Pretrained features are used across applications including fine-grained classification, detection, segmentation, and medical imaging.Source: §1 · p. 1
  • Foundation models face a source-target mismatch because target-task details may be unavailable during pretraining, motivating broadly applicable representations.The mismatch can include different numbers of target classes or available samples.Source: §1 · p. 1
  • The theoretical reasons transfer learning works between tasks in this setting remain relatively little understood..Source: §1 · p. 2

1.1 OTHER RELATED WORK

  • The paper uses neural collapse to explain transferability and proposes studying representations learned from one classification task for adaptation to unseen tasks with few samples.It reports that neural collapse generalizes to new samples and classes, and that a linear classifier can be trained with few samples.Source: §1 · p. 2, §1.1 · p. 2

2 PROBLEM SETUP

The setup models a target classification task drawn from a distribution over tasks and a separate source classification task used to learn a feature map. A target classifier is then trained on the learned features, while foundation-model training is agnostic to the target task.

2 PROBLEM SETUP

  • The target is a k-class classification task drawn from an unknown distribution over tasks, while the auxiliary source task has l classes.Samples are represented with one-hot labels, and class-conditional distributions describe inputs given each label.Source: §2 · pp. 2–3
  • The target classifier has the form h = g ◦ f, using a feature map learned on the source task and a classifier trained on target samples in feature space.The source classifier is likewise composed of a feature map and a classifier, trained on source data.Source: §2 · p. 3
  • A foundation model's feature map is trained without target-task information, unlike methods that optimize features using properties of the target task.The target-task properties may include its number of classes and samples per class.Source: §2 · p. 3

3 NEURAL COLLAPSE

Neural collapse describes same-class penultimate-layer features concentrating around their class mean, alongside a geometric arrangement of class means and a nearest-class-mean-like classifier. The paper uses a class-distance normalized variance measure to formalize a related form of collapse for its analysis.

3 NEURAL COLLAPSE

  • Neural collapse includes concentration of same-class features around their class mean, with centered class means asymptotically forming a scaled simplex equiangular tight frame and the last-layer classifier approaching a nearest-class-mean rule.Prior empirical observations concern terminal training stages under MSE and cross-entropy losses.Source: §3 · p. 4
  • Prior theory studies neural collapse in Gaussian mixtures, regularized MSE settings, and certain global optima of cross-entropy with weight decay.The cited work identifies settings in which collapse holds for training or at global minima.Source: §3 · p. 4
  • The paper defines class-distance normalized variance (CDNV) as a variant of within-class variance collapse for a feature map and two class-conditional distributions.Its definition is extended to finite sample sets, and decreasing within-class feature variance relative to interclass distance is expected to support classification.Source: §3 · p. 4

4 NEURAL COLLAPSE ON UNSEEN DATA

The analysis shows that neural collapse on source classes can extend to unseen samples and, under random class sampling and class-separation conditions, to new classes. It connects these results to low error for nearest-class-mean classification in few-shot settings.

4.1 GENERALIZATION TO NEW SAMPLES FROM THE SAME CLASS

  • For sufficiently large source-class training sets, small empirical CDNV implies small CDNV on unseen samples from those same classes.The population bound is at most (empirical CDNV + B)(1 + A)^2 with probability at least 1 − δ, where the additional terms account for generalization gaps.Source: §4.1 · p. 5

4.2 NEURAL COLLAPSE GENERALIZES TO NEW CLASSES

  • When source and target classes are sampled from a shared class distribution, low average source CDNV implies low expected CDNV between new classes if the source classes are representative.The bound can shrink as the number of source classes increases when feature maps keep distinct classes sufficiently separated.Source: §4.2 · p. 6

4.3 CDNV AND CLASSIFICATION ERROR

  • For a nearest empirical mean classifier, expected classification error is bounded in terms of CDNV, with tighter bounds under spherical symmetry and Gaussian features.Under the Gaussian assumption, the paper derives bounds that are exponentially small in feature dimension divided by CDNV.Source: §4.3 · p. 7
  • With high probability, target nearest-class-mean error is bounded by average source-training collapse plus terms that vanish as source sample and class counts increase.The result combines the same-class and new-class generalization bounds with the classification-error bound.Source: §4.3 · p. 7

5 EXPERIMENTS

The experiments examine whether neural collapse generalizes from source training data to held-out source samples and target classes, and how it relates to few-shot accuracy. Across datasets and architectures, stronger target-class collapse accompanies improved few-shot performance as the number of source classes increases.

5.2 RESULTS

  • The experiments found collapse on held-out source samples and target classes, with the pattern replicated across multiple datasets and architectures.Similar, weaker effects also appeared in feature maps from lower network layers.Source: §5 · p. 7, §5.2 · p. 10

5.1 SETUP

  • The protocol trains a source classifier, freezes its feature map, and fits a new target-task ridge-regression classifier without fine-tuning the feature map.Evaluation used randomly sampled 5-class tasks and averaged accuracy over 100 tasks.Source: §5.1 · p. 8, §5.2 · p. 9
  • Experiments used wide ResNets and convolutional networks across Mini-ImageNet, CIFAR-FS, FC-100, and balanced EMNIST.Results were averaged over 20 random initializations with 95% confidence intervals.Source: §5.1 · p. 8

5.2 RESULTS

  • The ridge-regression few-shot method was competitive with other methods on Mini-ImageNet, CIFAR-FS, and FC-100, especially at 1-shot, and achieved state of the art on FC-100.A learning-rate schedule and source-test-based model selection typically added a few percentage points.Source: §5.2 · p. 8
  • Compared with Distill-simple, the ridge-regression method underperformed its logistic-regression head but outperformed its nearest-neighbor classifier.The authors attribute the difference most likely to the choice of few-shot classifier.Source: §5.2 · p. 8
  • As source-class count increased, target-class CDNV indicated stronger collapse and few-shot accuracy consistently improved across the reported settings.Target-class CDNV remained relatively large compared with source-class CDNV, except on EMNIST.Source: Figure 1 · p. 9, §5.2 · pp. 9–10

6 CONCLUSIONS

The conclusion connects neural collapse to transfer learning with foundation models, reporting that collapse generalizes to held-out source samples and new classes drawn from the same distribution. When collapse emerges in new classes, few samples suffice to train an accurate linear classifier on the learned representation.

6 CONCLUSIONS

  • The paper concludes that neural collapse on new classes drawn from the source-class distribution can make a learned representation useful for accurate linear classification with very few samples.The authors present this as a justification for recent few-shot transfer-learning results.Source: §6 · p. 10

A EXPERIMENTAL DETAILS

The experimental details describe four datasets and the architectures used in the experiments. The following appendix section reports additional experiments for further insights.

A EXPERIMENTAL DETAILS

  • The experiments use Mini-ImageNet, CIFAR-FS, FC-100, and EMNIST, with source classes drawn from meta-training splits and target tasks from meta-test splits.EMNIST is split into 35 source classes and 12 target classes.Source: A EXPERIMENTAL DETAILS · p. 15
  • The study evaluates wide ResNets and vanilla convolutional networks that share an architecture except for the residual connections.The networks are denoted WRN-N-M and Conv-N-M, respectively.Source: A EXPERIMENTAL DETAILS · p. 15

B ADDITIONAL EXPERIMENTS

B.1 VARYING HYPERPARAMETERS

The appendix tests how neural-collapse measures and few-shot performance vary with training choices, feature-layer depth, class-mean distances, and the CCNV definition. It reports that target-class collapse tracks few-shot performance and that CCNV trends are qualitatively similar to CDNV trends.

B.1 VARYING HYPERPARAMETERS

  • Smaller learning rates tend to reduce source-data CDNV but increase target-class CDNV, and few-shot performance is generally worse in those cases.With learning-rate scheduling, more source classes improve few-shot performance while target-class CDNV tends to decrease; performance peaks near the target-CDNV minimum, then slightly declines as target CDNV rises during source overfitting.Source: §B.1 · pp. 15–16

B.2 NEURAL COLLAPSE AND LOWER LAYERS

  • The top embedding layer has lower CDNV and better 5-shot performance than the second-to-last layer, though collapse and associated few-shot performance persist in the lower layer less strongly.Source: §B.2 · p. 16

B.3 DYNAMICS OF THE CLASS-EMBEDDING DISTANCES

  • During training, minimum distances between empirical and true source-class means tend to increase, while target-class mean distances tend to grow with the number of source classes.Source: §B.3 · p. 16, Figure 7 · p. 21

B.4 CLASS-COVARIANCE NORMALIZED VARIANCE

  • CCNV decreases on source train and test data during training, and target-class CCNV decreases with more source classes while source-class CCNV increases; these trends qualitatively resemble CDNV.The appendix also reports CCNV and accuracy across source, test, and target data for WRN-28-4 on Mini-ImageNet and Conv-28-4 on CIFAR-FS.Source: §B.4 · pp. 18–19, Figure 8 · p. 21, Figure 9 · p. 22

C PROOF OF PROPOSITION 1

The proof of Proposition 1 relates population class-pair variation to its empirical counterpart with high probability. Its intermediate steps target empirical control of feature moments and class-mean distances.

C PROOF OF PROPOSITION 1

  • The proof targets upper bounds on population feature second moments and lower bounds on squared population feature means using their sample estimates.Source: C PROOF OF PROPOSITION 1 · p. 24
  • With probability at least 1 − δ, Proposition 1 bounds population pairwise variation by empirical pairwise variation with multiplicative and additive terms A and B.Source: C PROOF OF PROPOSITION 1 · p. 24
  • The argument uses triangle-inequality bounds to relate empirical and population class-mean distances before combining them with variation estimates.Source: C PROOF OF PROPOSITION 1 · p. 25

D PROOF OF PROPOSITIONS 2

The proof of Proposition 2 considers a finite function set whose class means have positive minimum separation. It uses a complexity-based argument and relates its statistic to average pairwise class variation.

D PROOF OF PROPOSITIONS 2

  • Proposition 2 assumes a finite function set with positive minimum separation between class means.Source: D PROOF OF PROPOSITIONS 2 · p. 26
  • The proof invokes Maurer and Pontil’s theorem and introduces partial-difference operators and maximal differences for its argument.Source: D PROOF OF PROPOSITIONS 2 · p. 26
  • The proof identifies its statistic A(Hf(P)) with the average pairwise class variation, then relates its expectation to variation over class distributions.Source: D PROOF OF PROPOSITIONS 2 · p. 26

E ANALYSIS FOR SECTIONS 4.1-4.2

The analysis studies ReLU feature maps through spectral complexity and derives concentration results for their first and second moments. It uses a canonical network representation with normalized hidden-layer norms in the analysis.

E ANALYSIS FOR SECTIONS 4.1-4.2

  • A ReLU feature map's spectral complexity is defined using layer norms and upper bounds its Lipschitz constant.The analysis fixes network depth and layer dimensions; the stated complexity uses the spectral norms of the weight matrices.Source: E ANALYSIS FOR SECTIONS 4.1-4.2 · p. 29
  • The analysis represents each network equivalently with hidden-layer matrices normalized to norm 1 and a final-layer norm equal to its spectral complexity.This canonical form follows from positive homogeneity of ReLU.Source: E ANALYSIS FOR SECTIONS 4.1-4.2 · p. 29
  • The first and second moments of ReLU feature maps with bounded spectral complexity concentrate around their means.The passage connects this result to bounds on terms used in the analysis of unseen examples.Source: E ANALYSIS FOR SECTIONS 4.1-4.2 · p. 29

F ANALYSIS FOR SECTION 4.2

Lemma 2 considers iid uniform points in a p-dimensional unit cube and uses a constant independent of sample size and dimension. Its proof uses pairwise-distance probabilities, a union bound, and a threshold condition.

F ANALYSIS FOR SECTION 4.2

  • Lemma 2 studies iid uniform points in the p-dimensional unit cube and introduces a positive constant independent of n and p.The passage's statement is truncated before the quantity being bounded is shown.Source: F ANALYSIS FOR SECTION 4.2 · p. 33
  • The proof bounds pairwise-distance probabilities using the volume of a p-dimensional ball and then applies a union bound over point pairs.It compares ball volume with the unit cube's volume.Source: F ANALYSIS FOR SECTION 4.2 · p. 33
  • The proof's function h(n, D) is positive exactly when D is below M^-1/p, with M defined from the number of point pairs.The subsequent bound invokes Ramanujan's approximations of the Gamma function.Source: F ANALYSIS FOR SECTION 4.2 · p. 34

G ANALYSIS FOR SECTION 4.3

The analysis bounds nearest-mean classification error using the average class-distance normalized variance (CDNV). It considers balanced classes and gives cases for general, spherically symmetric, and Gaussian feature distributions.

G ANALYSIS FOR SECTION 4.3

  • For balanced k-class problems, nearest-mean classification error is upper-bounded in terms of average pairwise CDNV; small average CDNV indicates expected small error.The classifier assigns an example to the class whose feature mean is nearest.Source: G ANALYSIS FOR SECTION 4.3 · p. 34
  • The analysis treats general, spherically symmetric, and spherical Gaussian feature distributions, with the Gaussian case assuming equal variances and class means spanning a regular simplex.The symmetric-case discussion also specifies a value for s(f, P), while the general case may use V_max.Source: G ANALYSIS FOR SECTION 4.3 · pp. 34–35

H LIST OF NOTATION

  • The notation glossary distinguishes source and target distributions and defines CDNV as a class-distance normalized variance between class distributions.It also lists class counts, sample sizes, feature maps, and transfer risk.Source: H LIST OF NOTATION · p. 38

Related papers

Loading 2112.15121v2…