Source-linked AI summary
Neural collapse with unconstrained features
Dustin G. Mixon, Hans Parshall, Jianzong Pi
TL;DR
Neural collapse is observed after empirical-risk minimization, but its emergence is not explained by existing theory. The paper introduces an unconstrained-features model to study this question and finds that gradient descent can produce strong neural collapse through the model’s optimization landscape. The model is restricted to training-set features and therefore does not generalize to test data.
Problem
The paper asks why neural collapse emerges from empirical risk minimization, despite existing theory under-explaining the performance of neural-network training.
Method
The paper studies an unconstrained-features model and analyzes an invariant subspace of its gradient-descent dynamics.
Results
Gradient descent initialized near the origin empirically produces strong neural collapse, and rank W0 ≥ C − 1 guarantees its emergence asymptotically.
Takeaways & Limitations
The model provides an explanation for neural-collapse emergence in terms of the empirical-risk optimization landscape.
Takeaways & Limitations
The unconstrained-features model defines feature maps only on the training set and cannot generalize to a test set.
Abstract
from arXiv · showhide
Neural collapse is an emergent phenomenon in deep learning that was recently discovered by Papyan, Han and Donoho. We propose a simple "unconstrained features model" in which neural collapse also emerges empirically. By studying this model, we provide some explanation for the emergence of neural collapse in terms of the landscape of empirical risk.
1 Introduction
Neural collapse describes geometric regularities observed after empirical-risk minimization, including within-class variability collapse, simplex ETF class means, self-duality, and nearest-class-center classification. The paper addresses why this structure emerges and notes its potential generalization benefit when features generalize.
- Observed phenomenon: Neural collapse emerges in the terminal phase after the classifier interpolates the training set.The observed properties arise when empirical risk continues to be minimized after training-set interpolation.
- Neural-collapse properties: Within-class variability collapses: every training example of class c shares a class mean µc.
- Neural-collapse properties: The centered class means form a simplex equiangular tight frame, giving them highly symmetric geometry.The simplex geometry is linked to optimal spherical coding and symmetric Voronoi cells.
- Neural-collapse properties: Neural collapse includes self-duality between normalized classifier weights and normalized centered class means.
- Neural-collapse properties: The classifier is equivalent to assigning a feature to its nearest class center.Regular-simplex class means induce symmetric Voronoi cells that support this equivalence.
- Motivation and open question: If the feature map generalizes, simplex-organized features can help the classifier generalize because perturbed test features remain likely within the correct class Voronoi cell.The paper presents this as a theoretical justification for neural collapse’s machine-learning relevance.
- Motivation and open question: The paper studies why neural collapse emerges from empirical risk minimization using an unconstrained-features model and an invariant gradient-descent subspace.The model empirically exhibits strong neural collapse, while the dynamical analysis explains its emergence through the optimization landscape.
2 Unconstrained features and strong neural collapse
The unconstrained-features model treats training-set feature values as freely variable, making neural-collapse dynamics analytically tractable but eliminating ordinary test-set generalization. Gradient descent initialized near the origin empirically produces strong neural collapse.
- Unconstrained-features model: The model represents training-set features as a matrix H whose columns are the feature vectors h(xi).Because inputs are distinct, the feature map restricted to the training set can be identified with Rp×CN.
- Unconstrained-features model: Feature maps are defined only on the training set, so this model has no hope of generalizing to a test set.This deliberate restriction facilitates the study of neural collapse.
- Optimization behavior: Gradient descent is initialized close to the origin, from which local optimization produces the phenomena called strong neural collapse.The phenomena are illustrated in Figure 1.
C 1C. (SNC3)
Strong neural collapse is empirically approached by gradient descent from near-origin initialization, and it forms a subset of global empirical-risk minimizers that implies neural collapse.
- Strong neural collapse points are global minimizers of empirical risk.
- Strong neural collapse achieves zero empirical risk.
- Strong neural collapse implies the standard neural-collapse properties.
- Gradient descent from near-origin initialization approaches strong neural collapse in the limit.The experiment initializes H and W near the origin and tracks relative errors for the strong-collapse conditions.
3 The effect of empirical risk minimization
The analysis identifies an invariant subspace that gradient dynamics approximately follow and shows that, under a rank condition, trajectories converge to strong neural collapse. It further characterizes strong-neural-collapse points as exactly the global minimizers within the model.
- Gradient dynamics: Gradient flow can be analyzed through a linearized system whose operators are self-adjoint and simultaneously diagonalizable over five eigenspaces.The paper decomposes the dynamics by orthogonal projections onto these eigenspaces.
- Invariant subspace: Figure 2 empirically indicates that the full gradient-descent trajectory remains at small relative distance from S.The distance is measured as ∥Z − ΠSZ∥E/∥Z∥E for C = N = 3 and p = 15.
- Gradient dynamics: Theorem 2 shows that gradient flow initially magnifies the component in T while driving b toward span{1C}.Consequently, the initial trajectory approximately follows the invariant subspace S.
- Invariant subspace: If the initialization lies in S and rank W0 ≥ C − 1, the trajectory converges to a limit exhibiting strong neural collapse.The invariant-subspace constraints are preserved by the dynamics.
- Landscape of empirical risk: The global minimizers of the empirical risk are precisely the points that exhibit strong neural collapse.Within S, equality in the nonnegative risk bound occurs exactly when the strong-neural-collapse conditions hold.
4 Discussion
The paper introduces an unconstrained-features model that captures observed neural-collapse phenomena and identifies an invariant subspace encouraging their emergence. It leaves the full dynamics, initialization dependence, alternative optimizers, and broader model classes for future study.
- Contributions: The paper’s model captures observed neural-collapse phenomena and identifies an invariant subspace of gradient descent that encourages neural collapse.These are presented as the paper’s main contributions.
- Open questions: Future work includes characterizing full gradient-flow dynamics, initialization dependence, alternative optimizers such as Adam, and other theoretically tractable models.The discussion explicitly lists these open directions.