PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training

Cong Fang, Hangfeng He, Qi Long, Weijie J. Su

arXiv:2101.12699v3cs.LGcs.CVmath.OCstat.ML

TL;DR

Developing a theoretical account of deep learning is difficult because of the intricate interactions among network layers. The paper introduces the Layer-Peeled Model as a tractable surrogate; it reproduces neural collapse and predicts an unknown phenomenon in imbalanced training.

  • Problem

    Intricate nonconvex interactions among neural-network layers make it difficult to build a theoretical foundation and pinpoint how layers influence outputs.

    Source: §1 · p. 1

  • Method

    The paper proposes the Layer-Peeled Model, a nonconvex surrogate that isolates last-layer classifiers and activations to make their interaction more amenable to analysis.

    Source: §1 · pp. 1–2

  • Results

    Neural collapse occurs in the Layer-Peeled Model, which also predicts a previously unknown phenomenon in imbalanced-data training.

    Source: §1 · p. 1, A. Two Applications · p. 2

  • Takeaways & Limitations

    A mathematically tractable surrogate that retains neural-network characteristics offers a practical alternative for reasoning about deep learning.

    Source: §1 · p. 1

Abstract

from arXiv · show

In this paper, we introduce the \textit{Layer-Peeled Model}, a nonconvex yet analytically tractable optimization program, in a quest to better understand deep neural networks that are trained for a sufficiently long time. As the name suggests, this new model is derived by isolating the topmost layer from the remainder of the neural network, followed by imposing certain constraints separately on the two parts of the network. We demonstrate that the Layer-Peeled Model, albeit simple, inherits many characteristics of well-trained neural networks, thereby offering an effective tool for explaining and predicting common empirical patterns of deep learning training. First, when working on class-balanced datasets, we prove that any solution to this model forms a simplex equiangular tight frame, which in part explains the recently discovered phenomenon of neural collapse \cite{papyan2020prevalence}. More importantly, when moving to the imbalanced case, our analysis of the Layer-Peeled Model reveals a hitherto unknown phenomenon that we term \textit{Minority Collapse}, which fundamentally limits the performance of deep learning models on the minority classes. In addition, we use the Layer-Peeled Model to gain insights into how to mitigate Minority Collapse. Interestingly, this phenomenon is first predicted by the Layer-Peeled Model before being confirmed by our computational experiments.

1. Introduction · A. Two Applications.

The Layer-Peeled Model simplifies deep-network analysis by isolating the top layer, and it captures neural collapse on balanced data while predicting minority-class collapse under imbalance. Experiments on deep networks confirm the predicted minority-collapse transition.

1. Introduction.

  • Deep-network theory is difficult because nonconvex optimization couples layers, and this challenge grows with network depth.The paper proposes a tractable surrogate that retains neural-network characteristics and explains an observed pattern while predicting a new one.Source: §1 · p. 1
  • The Layer-Peeled Model remains nonconvex but is more amenable to analysis because it reduces the interaction to the top layer and its feature variables.Its purpose is to provide a practical, mathematically tractable surrogate for quantitative patterns in deep neural networks.Source: §1 · pp. 1–2

A. Two Applications.

  • The model peels off the top layer and represents bottom-layer activations for each training example as decision variables under an overall ℓ2-norm constraint.This top-down approach models all bottom layers collectively as one variable.Source: A. Two Applications · p. 2
  • For equal class sizes, neural collapse occurs among global minimizers of the Layer-Peeled Model, for loss functions including cross-entropy and contrastive loss.Neural collapse includes within-class feature concentration, equally spaced centered class means, and classifiers aligned with those means.Source: A. Two Applications · p. 2, §1 · p. 2
  • The imbalanced-data analysis considers K_A majority classes with n_A examples each and K_B minority classes with n_B examples each, defining R = n_A/n_B > 1.The minority count is K_B = K − K_A.Source: A. Two Applications · p. 2
  • The Layer-Peeled Model predicts, and deep-learning experiments confirm, that minority classifiers collapse to one vector once the imbalance ratio exceeds a threshold.The prediction closely matches the experimental phase-transition point.Source: A. Two Applications · pp. 2–3, Figure 2 · p. 3
  • Minority collapse occurs at a finite imbalance ratio; even below the transition, minority classifiers are closer together than majority classifiers, theoretically and empirically.Below the threshold, minority classes remain distinguishable by their classifiers; above it, they become indistinguishable.Source: A. Two Applications · p. 3

B. Related Work. · 2. Derivation

The paper situates its work in neural-collapse research, then heuristically derives the Layer-Peeled Model by isolating the top layer and approximating the feasible feature range. The derivation is nonrigorous, and the paper contrasts its model with prior surrogates and notes that replacing the ℓ2 constraint prevents neural collapse.

B. Related Work.

  • Prior theoretical work on deep learning includes reviews, while neural collapse by (1) is highlighted for its mathematically elegant and convincing insights.The paper points to works (15–29), reviews (30–33), and references therein.Source: B. Related Work · p. 3
  • On balanced datasets, neural collapse comprises within-class feature variation vanishing, centered class means forming a simplex ETF, classifiers aligning with class means, and nearest-mean decisions.These four properties are attributed to (1).Source: B. Related Work · p. 3
  • With cross-entropy and ℓ2 regularization, neural-collapse symmetry and stability were shown to improve generalization, robustness, and interpretability during terminal-phase benign overfitting.The reported setting is when the trained model interpolates the in-sample training data; the claims are attributed to (1).Source: B. Related Work · p. 3
  • Concurrent surrogates also produced neural collapse, but (41, 42) used per-class norm constraints, (40) used unconstrained features and squared loss, and (43) used convex duality.The paper describes (43) as observing neural collapse in a ReLU-based model via strong duality under certain conditions.Source: B. Related Work · p. 3

2. Derivation

  • The derivation isolates the top-layer weights, treats last-layer features as decision variables, and assumes their feasible range under a norm constraint is approximately an ellipse.The authors motivate the feature-space assumption by the dual relationship between features and weights under ℓ2 constraints.Source: §2 · pp. 3–4
  • The Layer-Peeled Model is a nonconvex analytical surrogate derived by approximating the feasible feature range, with weight decay preventing the cross-entropy objective from attaining its minimum at infinity.The derivation omits network biases; the model’s data dependence is implicitly incorporated into its feature constraint threshold.Source: §2 · pp. 3–4
  • Neural collapse does not emerge when the model’s second constraint uses an ℓq norm for q ≠ 2, which partly justifies the ℓ2 feature-range assumption.The paper states this result formally as Proposition 2 in Section 6.Source: §2 · p. 4

3. Layer-Peeled Model for Explaining Neural Collapse · A. Cross-Entropy Loss.

For balanced datasets with p ≥ K−1, the Layer-Peeled Model with cross-entropy loss reproduces neural collapse: every global minimizer has within-class feature collapse and simplex-ETF class geometry. Its minimizers also satisfy the paper’s neural-collapse properties NC1–NC4.

3. Layer-Peeled Model for Explaining Neural Collapse

  • The balanced-dataset analysis assumes p ≥ K−1 and finds that the Layer-Peeled Model displays neural collapse, as in deep-learning training.The paper notes that this dimensionality assumption holds in many popular network architectures.Source: §3 · p. 4

A. Cross-Entropy Loss.

  • The model uses cross-entropy loss with features set to W h_k,i and offers an almost closed-form expression for any global minimizer.The paper presents this as an analytically tractable alternative to treating complex deep networks as black boxes.Source: A. Cross-Entropy Loss · p. 4
  • Every global minimizer in the balanced case has class-mean features forming a K-simplex ETF, with within-class features identical.The result applies to Eq. (7) with cross-entropy loss; the ETF is specified up to scaling.Source: A. Cross-Entropy Loss · pp. 4–5
  • Minimizers are equivalent up to rotation because simplex ETFs are invariant under rational transformations.The passage refers to the rational invariance of simplex ETFs.Source: A. Cross-Entropy Loss · p. 4
  • The minimizer’s feature and classifier geometry satisfies NC1–NC3, while its prediction rule also satisfies NC4.For a given last-layer feature h, the predicted class is arg max_k w⋆_k · h.Source: A. Cross-Entropy Loss · p. 5
  • The model’s neural-collapse result is presented as evidence of its effectiveness for analyzing neural networks.The paper also notes that other models were proposed to justify neural collapse.Source: A. Cross-Entropy Loss · p. 5

B. Extensions to Other Loss Functions.

The Layer-Peeled Model extends neural-collapse analysis beyond cross-entropy: in the balanced case, it yields simplex-ETF features for supervised contrastive loss and neural-collapse solutions for softmax-based losses under stated conditions.

Contrastive Loss

  • For supervised contrastive loss, every global minimizer has last-layer features forming a K-simplex ETF.The theorem’s geometry corresponds to pulling same-class examples together and pushing between-class inner products toward their minimum.Source: B. Extensions to Other Loss Functions · p. 5

Softmax-Based Loss

  • The softmax-based loss family includes cross-entropy and, for q > 1, a loss using g1(x) = (1 − x)^q and g2(x) = x^q.The theorem’s first part guarantees the Eq. (9) solution is a global minimizer but does not rule out other solutions.Source: B. Extensions to Other Loss Functions · p. 5
  • For balanced data, the softmax-based Layer-Peeled Model has the Eq. (9) solution as a global minimizer; every global minimizer has this form under strict-convexity and monotonicity conditions.Specifically, g2 must be strictly convex and at least one of g1 or g2 strictly monotone; neural collapse therefore continues under these regularity conditions.Source: B. Extensions to Other Loss Functions · p. 5
  • Experimental confirmation of neural collapse with these loss functions is left for future work.Source: B. Extensions to Other Loss Functions · p. 5

4. Layer-Peeled Model for Predicting Minority Collapse · A. Technique: Convex Relaxation. · B. Minority Collapse.

Using a convex relaxation of the Layer-Peeled Model, the paper analyzes how class imbalance affects long-trained networks and predicts Minority Collapse, in which minority classifiers become indistinguishable and impair minority-class prediction.

4. Layer-Peeled Model for Predicting Minority Collapse.

  • Class imbalance is associated with inferior deep-learning performance, yet its effects lack a solid theoretical footing; this section uses the Layer-Peeled Model to characterize them for sufficiently long training.The paper states that neural collapse no longer emerges with class imbalance and identifies Minority Collapse as a performance limit, especially for minority classes.Source: §4 · pp. 5–6

A. Technique: Convex Relaxation.

  • The convex relaxation preserves the Layer-Peeled Model’s solutions under stated conditions, including p ≥ 2K, convexity of the loss, and saturation of the first K diagonal constraints.The resulting program’s size is independent of the number of training examples, and the lemma implies last-layer features collapse to class means, satisfying Property (NC1).Source: A. Technique: Convex Relaxation · p. 6
  • The relaxation’s assumptions cover many losses; the diagonal-constraint condition is proved for cross-entropy, and a nuclear-norm-constrained convex relaxation is another option.The paper motivates the relaxation by the lack of a simple minimizer expression for the imbalanced nonconvex model, which complicates numerical solution.Source: A. Technique: Convex Relaxation · p. 6

B. Minority Collapse.

  • Above a finite imbalance threshold, minority classifiers collapse to a single vector, with zero between-class angles and approximately equal classifier lengths.Figure 3 shows the average cosine between minority classifiers reaching 1 beyond a threshold; below it, the average between-minority-class angle decreases as imbalance rises.Source: Figure 3 · p. 6, B. Minority Collapse · p. 7
  • The phase-transition threshold R0 depends on KA, KB, EH, and EW, and decreases when KB, EH, or EW decreases with the other parameters fixed.The reported numerical examples for this dependence are in Figure S2 of the SI Appendix.Source: B. Minority Collapse · p. 7
  • After Minority Collapse, the network predicts equal probabilities for all minority classes regardless of input, making its minority-conditioned predictive ability no better than a coin toss.The passage also warns that minority predictions may remain close to one another even before collapse when the imbalance ratio is large.Source: B. Minority Collapse · p. 7
  • Under cross-entropy loss, with p ≥ K and fixed KA and KB, Theorem 5 establishes Minority Collapse as the imbalance ratio R tends to infinity.The theorem’s stated regime is nA/nB → ∞; the supplied theorem passage does not include its displayed conclusion.Source: B. Minority Collapse · p. 7
  • As imbalance grows, majority classes dominate the risk, encouraging majority-class angles to grow while shrinking between-minority-class angles toward zero.The paper notes that proving exact equality of minority classifiers at sufficiently large R remains a future question.Source: B. Minority Collapse · p. 7

C. Experiments. · 5. How to Mitigate Minority Collapse?

Experiments confirm that Minority Collapse occurs in deep networks and coincides with impaired minority-class test performance at extreme imbalance. The paper also finds that oversampling can mitigate collapse and improve test accuracy at selected rates, while very large rates may hurt performance.

C. Experiments.

  • Experiments train VGG and ResNet on FashionMNIST and CIFAR10, testing class splits (KA, KB) = (3, 7), (5, 5), and (7, 3).VGG dropout is replaced with batch normalization; models use cross-entropy and SGD with momentum 0.9, weight decay 5 × 10−4, and 350 epochs.Source: C. Experiments · p. 7
  • At R = 1000, minority-class test performance deteriorates when Minority Collapse has occurred or is about to occur, despite the fixed minority-class training examples.The comparison uses 6 minority examples per FashionMNIST class or 5 per CIFAR10 class at both imbalance levels.Source: C. Experiments · p. 7, Figure 5 · p. 8
  • Minority Collapse prevents zero training error because minority predictions are uniform, but minority examples form only a small share of training data, keeping total training error small.The paper attributes collapse to the Layer-Peeled Model constraints or neural-network ℓ2 regularization; lowering the penalty alone cannot eliminate it for arbitrarily large imbalance.Source: C. Experiments · pp. 7–8
  • Minority-class classifier angles collapse toward zero at sufficiently large imbalance, while their norms remain nearly equal; the effect worsens with more majority classes.Figure 4 also reports average cosine reaching 1 for ResNet18 with KA = 7 at R = 5000 on CIFAR10 and R = 3000 on FashionMNIST.Source: C. Experiments · p. 7, Figure 4 · p. 8, Table 1 · p. 8

5. How to Mitigate Minority Collapse?

  • In real-world models, increasing the oversampling rate enlarges angles between minority classifiers and makes Minority Collapse less detrimental to training accuracy.The experiment uses the Figure 5 setup.Source: Figure 6 · p. 9, §5 · p. 9
  • In the Layer-Peeled Model, oversampling acts as if each minority class had wrnB examples; when wr = R, all classifier-pair angles are equal.Proposition 1 establishes a global minimizer for the oversampling-adjusted model under its stated convex-loss and dimensionality conditions.Source: §5 · p. 9
  • Oversampling improves test accuracy on both minority classes and all classes at selected rates, but very large rates can degrade test performance.Among rates 1, 10, 100, and 1000, the best accuracy is never achieved at wr = 1000.Source: Table 2 · p. 9, §5 · p. 9

6. Discussion

The Layer-Peeled Model helps explain neural collapse on balanced datasets and predicts Minority Collapse above an imbalance threshold, a pattern also observed in computational experiments. The discussion identifies support for its ℓ2-norm ansatz, practical uses, and directions for extending the model.

Discussion

  • The Layer-Peeled Model explains neural collapse on balanced datasets and predicts that minority-class classifiers collapse to one vector above an imbalance threshold, as experiments confirm.The model isolates the last layer from the rest of the network.Source: §6 · p. 9
  • For K ≥3 and p ≥K, Proposition 2 shows that any global minimizer with q ∈(0, 2) ∪(2, ∞) fails Eq. (9), supporting ℓ2 as the only ℓq norm consistent with observed neural collapse.The proposition, together with Theorem 1, indirectly justifies the ansatz's ℓ2 norm.Source: §6 · pp. 9–10
  • Future work could quantify the model's gap from trained networks across architectures, account for sparse ReLU features, and test whether retaining multiple top layers captures new training patterns.The proposed m-Layer-Peeled Model retains the top m layers; the discussion specifically asks whether m = 2 captures new patterns.Source: §6 · p. 10
  • The model and its convex relaxation offer efficient ways to identify and mitigate imbalance-induced bias, while open questions include generalizing Minority Collapse, choosing loss functions, and designing fairness-aware sampling.The loss-function question follows the paper's findings in Section 5; fairness-aware sampling is proposed as a possible use case.Source: §6 · p. 10
  • More broadly, making part of a network modular can simplify empirical models, and a top-down strategy can consolidate prior findings and reveal new patterns in deep learning.The authors suggest the modeling strategy may support future research because of its mathematical convenience.Source: §6 · p. 10

A. Balanced Case. · A.1. Proofs of Theorem 1 and Proposition 2 .

The proofs establish the balanced-case global optimum and examine how the optimizer changes under alternative norm and binary-class settings. They also compare the predicted binary weight patterns with simulations and state an unresolved conjecture for q ≤ 1.

A.1. Proofs of Theorem 1 and Proposition 2 .

  • The proof reduces minimizing Eq. (7) to minimizing a symmetric quadratic function under suitable conditions.This is the key step in the direct global-minimum argument.Source: §A.1 · p. 12
  • Theorem 1’s proof establishes that Eq. (7) has global minimum L0, with Eq. (9) as the unique minimizer form.The proof determines the global minimum directly because the objective is nonconvex and KKT conditions are not sufficient.Source: §A.1 · pp. 12–13
  • For q ≠ 2, Proposition 2’s feasible symmetric solutions collapse each class’s features to its mean and align each weight with its feature, but their weights are not ETF vertices.The proposition’s proof shows that the minimizing parameter a differs from K − 1 unless q = 2.Source: §A.1 · p. 14
  • For binary classification with q ∈ (1, 2) ∪ (2, ∞), global minimizers retain Eq. (9)’s form but are not rotationally invariant.This contrasts with a K-simplex ETF, which is rotationally invariant.Source: §A.1 · p. 14
  • For binary classification, q > 2 yields parallel features and weights, whereas 1 < q < 2 yields one nonzero entry for each.The q > 2 pattern is specified as a signed all-ones vector; the q < 2 pattern is sparse.Source: §A.1 · p. 14
  • Simulations show that binary last-layer weights match neither predicted pattern, supporting ℓ2 as the best ℓq choice for modeling deep networks with the Layer-Peeled Model.The comparison is with the all-ones and sparse patterns in Lemma 3.Source: §A.1 · p. 15
  • For q ≤ 1, the authors conjecture that an ℓq regularizer would also produce sparse solutions, leaving this case for future work.Source: §A.1 · p. 15

A.2. Proofs of Theorems 3 and 4.

Theorem 3’s equality conditions yield Eq. (13), and conversely that condition attains equality. Theorem 4 establishes Eq. (9) as a global minimizer and, under stated strictness assumptions, characterizes all solutions.

Proof of Theorem 3

  • Theorem 3’s equality conditions imply [h1, . . . , hK]⊤[h1, . . . , hK] = EH and hence Eq. (13); conversely, Eq. (13) attains equality.Source: §A.2 · p. 17

Proof of Theorem 4

  • The lower bound L0 is attained, making it the global minimum of Eq. (7); equality in Eq. (56) is reachable when (H, W) satisfies Eq. (9), which is a minimizer.Source: §A.2 · p. 18

Lemma 4

  • With L defined in Eq. (54), and S1 and S2 as defined in (i) and (ii), respectively, equality in Eq. (59) holds if and only if |S1| = 0.Source: §A.2 · p. 19

B. Imbalanced Case. · B.1. Proofs of Lemma 1 and Proposition 1.

The proof reduces the original program to a convex program and uses a minimizer of that convex program to construct a minimizer of the original program. It also characterizes when all original minimizers have the form in Eq. (16), and states that Proposition 1 follows by the same argument.

B.1. Proofs of Lemma 1 and Proposition 1.

  • A feasible solution of the original program yields an X that satisfies the convex program’s constraints and is therefore feasible for Eq. (15).The proof first establishes X ⪰ 0 and verifies the other constraints.Source: §B.1 · p. 21
  • A minimizer X⋆ of Eq. (15) defines a feasible pair (H⋆, W⋆) for the original program through Eq. (16).The proof uses this construction to relate the two programs’ minima.Source: §B.1 · p. 21
  • The proof concludes that L0 is the global minimum of Eq. (7) and that the constructed pair (H⋆, W⋆) is a minimizer.It combines the objective comparisons in Eqs. (64) and (65).Source: §B.1 · p. 21
  • Eq. (64) implies X′ is a minimizer of Eq. (15); if (H′, W′) cannot be written as Eq. (16) with X⋆=X′, then for some k′∈[K], distinct i,j∈[n_{k′}] satisfy h_{k′,i}≠h_{k′,j}.Source: §B.1 · p. 22
  • When the diagonal entries satisfy sum_k X⋆(k,k) = E_H, all solutions of Eq. (7) have the form in Eq. (16).This condition completes the proof’s characterization of the solutions.Source: §B.1 · p. 22
  • The proof of Proposition 1 is omitted because it follows by the same argument.Source: §B.1 · p. 22

B.2. Proof of Theorem 5.

The proof of Theorem 5 uses a partial-class limit and contradiction argument. Its key lemma characterizes minimizers by zero features for the B-class examples and a minimum depending only on class counts and norm constraints.

B.2. Proof of Theorem 5.

  • For any minimizer (H, W) of Eq. (66), hk,i = 0p for all k in [KA + 1 : K] and i in [nB]; the global minimum L0 depends only on KA, KB, EH, and EW.Source: §B.2 · p. 22
  • The sequence-indexed objective minima LA and LB depend only on KA, KB, EH, and EW, not on ℓ.The proof applies Lemma 5 to the two separated optimization programs before comparing their objective values.Source: §B.2 · p. 23
  • The contradiction rules out a fixed positive limiting separation between weights for classes in [KA + 1 : K], establishing Theorem 5.The objective gap is bounded below by a positive quantity depending on ε, KA, KB, EH, and EW, but independent of the sequence index ℓ.Source: §B.2 · p. 23

PnA

The PnA analysis establishes L2 as the global minimum of Eq. (66) and shows that minimizers have zero features for minority-class indices; this does not imply all their network activations die.

PnA

  • Every minimizer has h_{k,i}=0 for minority-class indices k∈[KA+1:K] and i∈[nB].Equality in the lower bound requires zero features for k∈[KA+1:K].Source: PnA · pp. 25, 27
  • The analyzed cases establish L2 as the global minimum of Eq. (66), with feasible constructions attaining the lower bound.The proof combines three cases and gives a construction achieving equality in the lower bound.Source: PnA · pp. 25–26
  • Zero features for minority-class indices do not imply that all network activations for those classes die, because the analysis omits bias terms.Source: PnA · p. 26

C. Additional Results. · D. Additional Experimental Results.

The additional results explain why oversampling can have lower update variance than weight adjusting and establish a convex relaxation of the model. Additional experiments report a phase transition and show that ImageNet-pretrained classifiers do not converge to a Simplex ETF.

C. Additional Results.

  • Under nA ≫ nB and 1 ≪ wr ≲ nA/nB, the expected-scale step sizes are comparable, and the weighted-adjusting update has a larger second moment.With similarly sized class-gradient norms and wr ≍ nA/nB, the analysis concludes that oversampling's update variance is potentially much smaller.Source: C. Additional Results · p. 28
  • Oversampling can yield potentially much smaller update variance than weight adjusting, offering a justification for its greater optimization stability.Both methods optimize the same objective but use different algorithms; the comparison assumes updates are on the same expected scale.Source: C. Additional Results · pp. 27–28
  • For p ≥ K and a loss convex in its first argument, a minimizer of the nuclear norm-constrained convex program maps to a minimizer of the original program.The construction uses a thin SVD of the convex-program minimizer and a partial orthogonal matrix.Source: C. Additional Results · pp. 28–29
  • For cross-entropy loss, any minimizer Z⋆ of the convex program satisfies ∥Z⋆∥∗ = √(EH EW).Source: C. Additional Results · p. 29

D. Additional Experimental Results.

Related papers

Loading 2101.12699v3…