PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training

Cong Fang, Hangfeng He, Qi Long, Weijie J. Su

arXiv:2101.12699v3cs.LGcs.CVmath.OCstat.ML

TL;DR

Deep neural networks lack a principled theoretical foundation because multilayer optimization is difficult to analyze. The paper introduces a tractable Layer-Peeled Model that reproduces neural collapse and predicts Minority Collapse under class imbalance.

  • Problem

    Deep learning theory lacks principled guidance because interactions among nonlinear layers make neural-network training difficult to analyze.

  • Method

    The Layer-Peeled Model isolates the topmost layer and represents the remaining layers collectively through constrained decision variables.

  • Results

    The model exhibits neural collapse on balanced data and predicts a previously unknown phenomenon, Minority Collapse, on imbalanced data.

  • Takeaways & Limitations

    The Layer-Peeled Model provides a mathematically tractable framework for analyzing deep-learning training patterns, including minority-class degradation under imbalance.

Abstract

from arXiv · show

In this paper, we introduce the \textit{Layer-Peeled Model}, a nonconvex yet analytically tractable optimization program, in a quest to better understand deep neural networks that are trained for a sufficiently long time. As the name suggests, this new model is derived by isolating the topmost layer from the remainder of the neural network, followed by imposing certain constraints separately on the two parts of the network. We demonstrate that the Layer-Peeled Model, albeit simple, inherits many characteristics of well-trained neural networks, thereby offering an effective tool for explaining and predicting common empirical patterns of deep learning training. First, when working on class-balanced datasets, we prove that any solution to this model forms a simplex equiangular tight frame, which in part explains the recently discovered phenomenon of neural collapse \cite{papyan2020prevalence}. More importantly, when moving to the imbalanced case, our analysis of the Layer-Peeled Model reveals a hitherto unknown phenomenon that we term \textit{Minority Collapse}, which fundamentally limits the performance of deep learning models on the minority classes. In addition, we use the Layer-Peeled Model to gain insights into how to mitigate Minority Collapse. Interestingly, this phenomenon is first predicted by the Layer-Peeled Model before being confirmed by our computational experiments.

1. Introduction · A. Two Applications.

The paper introduces the Layer-Peeled Model as a tractable surrogate for analyzing deep neural networks despite their multilayer nonconvexity. It uses this model to recover neural collapse on balanced data and predict Minority Collapse on imbalanced data.

  • 1. Introduction: Deep learning’s empirical success relies heavily on heuristics, motivating a more rigorous theoretical framework.The paper presents theory as a potential means to advance deep learning methodology.
  • 1. Introduction: Neural-network training is difficult to analyze because nonconvex interactions among many layers obscure each layer’s influence on the output.The difficulty is compounded by the growing number of layers.
  • 1. Introduction: The Layer-Peeled Model simplifies this structure by isolating the topmost layer and representing all lower-layer features with constrained decision variables.The resulting interaction is only between the last-layer classifiers and the feature variables.
  • A. Two Applications.: For balanced datasets, any global minimizer of the Layer-Peeled Model exhibits neural collapse.The result applies to a family of losses including cross-entropy and contrastive loss.
  • A. Two Applications.: For imbalanced datasets, minority-class classifiers collapse to a single vector when the imbalance ratio R is sufficiently large.The derivation addresses both the model’s nonconvexity and the asymmetry caused by unequal class sizes.
  • A. Two Applications.: The Minority Collapse prediction accurately captures the phase transition where the cosine between minority classifiers becomes 1.Experiments use VGG13 on CIFAR10 subsets with majority classes fixed at 5000 and weight decay λ = 5 × 10^-3, 5 × 10^-4.
  • A. Two Applications.: As R increases, minority classifiers transition from distinguishable to indistinguishable at a finite threshold.The Layer-Peeled Model predicts this phase transition, which also appears in computational experiments on deep neural networks.

B. Related Work. · 2. Derivation

The paper situates its Layer-Peeled Model among neural-collapse results and related surrogate models, then heuristically derives it by isolating the top layer and approximating the remaining feature range. The resulting program is nonconvex but analytically tractable, with data dependence encoded implicitly through a threshold and neural collapse tied specifically to an ℓ2 constraint.

  • B. Related Work.: Neural collapse on balanced datasets comprises within-class variability collapse, simplex ETF class means, classifier–mean alignment, and nearest-mean decisions.These are labeled NC1–NC4 in the related-work discussion.
  • B. Related Work.: Cross-entropy with ℓ2 regularization corresponds to weight decay, whose symmetry and stability benefits appear during terminal-phase benign overfitting when training data are interpolated.The discussion connects this experimental setup to established neural-collapse behavior.
  • B. Related Work.: Concurrent surrogate models also peel off the top layer, but some impose per-class norm constraints, while another uses unconstrained features and squared loss.The Layer-Peeled Model instead employs an overall constraint.
  • 2. Derivation: The derivation heuristically isolates the topmost-layer weights while retaining the neural-network optimization structure and omitting biases for simplicity.The authors explicitly describe the derivation as nonrigorous and intended to reduce complexity.
  • 2. Derivation: The top-layer formulation introduces last-layer features generated by all preceding weights, then uses Lagrangian duality to replace the original program with constraints involving positive constants C1 and C2.The formulation is subsequently rewritten using H as a decision variable representing the feature matrix.
  • 2. Derivation: Approximating the constrained feature range as an ellipse yields the Layer-Peeled Model, a nonconvex program that is nevertheless generally mathematically tractable.The ansatz treats H as an ℓ2-space variable because it is dual to W and ℓ2 spaces are self-dual.
  • 2. Derivation: The model’s apparent data independence is misleading because the training data enter implicitly through the threshold EH, and neural collapse fails when the second constraint uses ℓq for q ≠ 2.The latter observation is presented as partial justification for the elliptical ansatz.

3. Layer-Peeled Model for Explaining Neural Collapse · A. Cross-Entropy Loss.

For balanced data, the Layer-Peeled Model with cross-entropy loss has minimizers exhibiting neural collapse: within-class features coincide, class means form a simplex ETF, and classifiers are dual to features. The model also provides an analytically tractable framework whose neural-collapse behavior extends to several popular loss functions.

  • 3. Layer-Peeled Model for Explaining Neural Collapse: The balanced-data analysis assumes n_k=n for every class and p≥K−1, an assumption satisfied by many popular network architectures.The section studies deep networks trained on balanced datasets and defers proofs to the SI Appendix.
  • A. Cross-Entropy Loss.: With cross-entropy loss, any global minimizer of the Layer-Peeled Model admits an almost closed-form expression.The feature is defined as z=W h_k,i, while the Layer-Peeled Model remains easier to analyze than complex deep neural networks.
  • A. Cross-Entropy Loss.: The minimizer’s class-mean features form a K-simplex ETF, revealing highly symmetric geometry in the last-layer features and weights.The theorem characterizes global minimizers in the balanced case, with the simplex ETF specified in Definition 1.
  • A. Cross-Entropy Loss.: Equivalent minimizers differ only up to rotation because simplex ETFs are invariant under the relevant transformations.This rotational equivalence applies to the minimizers (W⋆, H⋆).
  • A. Cross-Entropy Loss.: All within-class last-layer features are identical, while class-mean features form a K-simplex ETF up to scaling.The resulting geometry is the neural-collapse phenomenon described by the theorem.
  • A. Cross-Entropy Loss.: The minimizer satisfies neural-collapse properties (NC1)–(NC4), including classifier-feature duality and prediction by arg max_k w⋆_k · h.The class relationships and prediction rule together establish all four stated neural-collapse properties.
  • A. Cross-Entropy Loss.: The presence of neural collapse supports the Layer-Peeled Model’s effectiveness as a tool for analyzing neural networks.The paper notes that other models were also recently proposed to justify neural collapse.
  • A. Cross-Entropy Loss.: The Layer-Peeled Model continues to exhibit neural collapse for some popular loss functions beyond cross-entropy.This extension addresses loss functions selected to reflect different problem characteristics in modern deep learning.

B. Extensions to Other Loss Functions.

The Layer-Peeled Model extends neural-collapse results beyond cross-entropy to supervised contrastive and broad softmax-based losses. Contrastive loss yields a K-simplex ETF in last-layer features, while softmax-based losses recover the same structure under mild regularity conditions.

  • Contrastive Loss: Supervised contrastive loss produces neural collapse in the Layer-Peeled Model despite using label information only implicitly.The loss pulls same-class embeddings together and pushes different-class embeddings apart without explicitly involving last-layer classifiers.
  • Contrastive Loss: Any global minimizer of the contrastive-loss model forms a K-simplex ETF in its last-layer features.Within-class features coincide, while between-class inner products are minimized, yielding simplex vertices up to scaling.
  • Softmax-Based Loss: For balanced data, the softmax-based loss family includes cross-entropy and other convex constructions such as g1(x) = (1 − x)^q and g2(x) = x^q for q > 1.The family is defined using a nonincreasing convex g1 and a nondecreasing convex g2 on (0, 1).
  • Softmax-Based Loss: For any loss in the softmax-based family, the Eq. (9) solution is a global minimizer; with additional strictness conditions, every global minimizer has that form.The conditions require g2 to be strictly convex and at least one of g1 or g2 to be strictly monotone.
  • Softmax-Based Loss: Neural collapse therefore persists for softmax-based losses under mild regularity conditions, although experimental confirmation remains future work.The result does not require convexity of the loss itself, but the paper leaves empirical validation for later work.

4. Layer-Peeled Model for Predicting Minority Collapse · A. Technique: Convex Relaxation. · B. Minority Collapse.

The Layer-Peeled Model uses a convex relaxation to characterize long-trained networks under class imbalance, predicting Minority Collapse: minority classifiers become identical beyond a finite imbalance level, limiting minority-class prediction.

  • 4. Layer-Peeled Model for Predicting Minority Collapse: Class imbalance is common and empirically degrades deep-learning performance, but its effects lack a solid theoretical explanation because training dynamics are complex.Examples include an approximately eightfold class-size disparity in Places2 and more than one hundredfold disparity in Ontonotes.
  • 4. Layer-Peeled Model for Predicting Minority Collapse: Under class imbalance, neural collapse no longer emerges, and the Layer-Peeled Model predicts Minority Collapse as a fundamental limitation on minority-class performance.The phenomenon is analyzed for networks trained sufficiently long and is supported by numerical evidence in Figure S2.
  • A. Technique: Convex Relaxation.: A convex optimization program relaxes the nonconvex Layer-Peeled Model through a semidefinite formulation, enabling computational analysis of imbalanced training.The relaxation is motivated by standard semidefinite relaxation of quadratically constrained quadratic programs.
  • A. Technique: Convex Relaxation.: The relaxation preserves the Layer-Peeled Model’s information under stated conditions and has program size independent of the number of training examples.The conditions include convexity of the loss in its first argument and saturation of the first K diagonal constraints; the latter holds for cross-entropy.
  • A. Technique: Convex Relaxation.: For KA majority classes of size nA and KB minority classes of size nB, the imbalance ratio is defined as R := nA/nB > 1.The convex program is solved numerically to study minority classifiers as R increases.
  • B. Minority Collapse.: As R increases, minority classifiers exhibit two phases: their average between-minority angle first shrinks, then reaches zero beyond a threshold R0.The phase transition point depends on KA, KB, EH, and EW, and becomes smaller when EW, EH, or KB decreases while other parameters remain fixed.
  • B. Minority Collapse.: At the second phase, minority classifiers have approximately equal lengths and collapse to a single vector, causing equal predicted probabilities for all minority classes.The model therefore has no better predictive ability than assigning equal probabilities across those minority classes.
  • B. Minority Collapse.: For cross-entropy minimizers with p ≥ K and fixed KA and KB, the theorem establishes Minority Collapse as R ≡ nA/nB →∞.The paper notes that Minority Collapse is initially a prediction of the Layer-Peeled Model whose occurrence in real-world neural networks remains an open question.

C. Experiments. · 5. How to Mitigate Minority Collapse?

Experiments confirm Minority Collapse in VGG and ResNet under severe class imbalance, where minority-class classifier angles collapse and minority test performance deteriorates. Oversampling enlarges minority-class angles and improves test accuracy at suitable rates, but excessively large rates can harm test performance.

  • C. Experiments.: Experiments on VGG and ResNet across FashionMNIST and CIFAR10 confirm Minority Collapse under sufficiently large imbalance ratios.The setups split 10 classes into majority and minority groups with combinations (KA, KB) = (3, 7), (5, 5), and (7, 3).
  • C. Experiments.: As R increases, the angles between minority classifiers collapse to zero while their classifier norms remain constant across minority classes.Together, these geometric patterns provide evidence for Minority Collapse in trained neural networks.
  • C. Experiments.: At R = 1000, Minority Collapse coincides with deteriorating minority-class test performance.Figure 5 evaluates minority-class accuracy using 6 or 5 examples per minority class for FashionMNIST or CIFAR10, respectively.
  • C. Experiments.: Minority Collapse prevents zero training error because uniform minority-class predictions cannot satisfy the argmax rule for minority examples.The resulting training error is mostly concentrated in minority classes but remains small because those examples constitute a small portion of the training set.
  • 5. How to Mitigate Minority Collapse?: Oversampling duplicates minority examples at rate wr, effectively changing each minority class from nB examples to wrnB in the Layer-Peeled Model.This transformation is formalized through the oversampling-adjusted Layer-Peeled Model and Proposition 1.
  • 5. How to Mitigate Minority Collapse?: Increasing wr enlarges angles between minority classifiers and makes Minority Collapse less detrimental to training accuracy in real-world networks.Figure 6 uses the same experimental setup as Figure 5 and corroborates the Layer-Peeled Model’s prediction.
  • 5. How to Mitigate Minority Collapse?: When wr = nA/nB ≡ R, all pairwise angles between last-layer classifiers become equal, including pairs involving majority and minority classes.Proposition 1 and Lemma 1 establish this conclusion for the oversampling-adjusted model.
  • 5. How to Mitigate Minority Collapse?: Oversampling improves minority-class and overall test accuracy at suitable rates, but very large wr can degrade test performance and does not fully resolve Minority Collapse.Table 2 reports the best accuracy among wr values 1, 10, 100, and 1000; the best result is never achieved at wr = 1000.

6. Discussion

The Layer-Peeled Model provides a tractable top-down framework for explaining neural collapse and identifying imbalance-induced bias, while motivating extensions and further validation against deep networks. Its analysis also supports the ℓ2 ansatz and highlights limitations and open questions for broader architectures and imbalance settings.

  • Modeling strategy: The Layer-Peeled Model isolates the last layer to provide an analytically and numerically tractable framework for understanding well-trained deep neural networks and neural collapse on balanced datasets.The model follows a top-down derivation that separates the last layer from the remaining layers.
  • Theoretical implications: The ℓ2 norm in the ansatz is the only ℓq-norm choice consistent with empirical observations.For q ∈ (0, 2) ∪ (2, ∞), global minimizers do not satisfy the relevant neural-collapse condition, so neural collapse does not emerge.
  • Limitations and extensions: Future work should quantify the gap between the Layer-Peeled Model and trained networks, including dependencies on architecture and sparsity from ReLU features.The discussion also proposes modifying the constraints to reflect architectures and testing whether m = 2 captures new training patterns.
  • Practical implications: The model and its convex relaxation offer efficient tools for identifying and mitigating class-imbalance bias, while leaving general imbalance and loss-function choices as open questions.The discussion specifically calls for extending Minority Collapse beyond two-valued class sizes and studying loss functions that mitigate it.
  • Modeling strategy: The paper argues that modularizing part of a formidable empirical network and modeling the top layers from the top down can consolidate prior findings.This modeling strategy is presented as broadly useful beyond the specific Layer-Peeled Model.

A. Balanced Case. · A.1. Proofs of Theorem 1 and Proposition 2 .

In the balanced case, the nonconvex Layer-Peeled Model is solved globally by reducing its objective to a symmetric quadratic function, yielding a unique minimizer form characterized by simplex ETF structure. The proofs further show that the binary K=2 case depends sharply on q, while simulations favor ℓ2 regularization for modeling neural networks.

  • A. Balanced Case.: The nonconvex objective is analyzed by proving that its global minimization is equivalent, under suitable conditions, to minimizing a symmetric quadratic function.The reduction is needed because KKT conditions are not sufficient for optimality.
  • A.1. Proofs of Theorem 1 and Proposition 2 .: Eq. (9) is the unique form of the global minimizers, and the corresponding objective value is the global minimum L0.The proof establishes both directions: solutions satisfying Eq. (9) attain equality, and equality forces that form.
  • A.1. Proofs of Theorem 1 and Proposition 2 .: The equality conditions imply the class features collapse to their means and each classifier weight is parallel to its class feature, but the weights form an ETF only when a = K −1.These properties are identified within the symmetric feasible set SR.
  • A.1. Proofs of Theorem 1 and Proposition 2 .: For K = 2 and q ∈(1, 2) ∪(2, ∞), global minimizers retain Eq. (9)'s form but are not rotationally invariant, unlike a K-simplex ETF.The binary analysis assumes one training example per class, with extension to multiple examples stated as direct.
  • A.1. Proofs of Theorem 1 and Proposition 2 .: When q > 2, both features and weights are parallel to one vector, whereas when 1 < q < 2, both are sparse with only one nonzero entry.These are the specific ETF-related structures established for the optimal solution of the binary program.
  • A.1. Proofs of Theorem 1 and Proposition 2 .: Figure 7 shows that binary neural-network last-layer weights exhibit neither the all-ones nor sparse patterns predicted by Lemma 3, supporting ℓ2 as the best modeling choice among ℓq norms.The conclusion is based on the reported simulation results for last-layer weights.

A.2. Proofs of Theorems 3 and 4.

The proofs establish the equality characterization for Theorem 3 and show that Eq. (9) globally minimizes Eq. (7). Under strict convexity and monotonicity assumptions, every minimizer has the form of Eq. (9).

  • Proof of Theorem 3: Theorem 3’s equality conditions imply [h1, . . . , hK]⊤[h1, . . . , hK] = EH, which in turn implies Eq. (13).The proof also verifies that equality is attainable whenever H satisfies Eq. (13).
  • Proof of Theorem 4: Eq. (9) is a global minimizer of Eq. (7), because it attains the lower bound on the objective established in the proof.The proof identifies the minimum as L0 and directly verifies attainability under Eq. (9).
  • Proof of Theorem 4: Under strict convexity of g2 and strict monotonicity of g1 or g2, all solutions minimizing Eq. (7) satisfy Eq. (9).The equality conditions from the convexity and monotonicity steps are combined with Lemma 2 to establish uniqueness in the stated form.
  • Proof of Lemma 4: Lemma 4 shows that the lower-bound expression is maximized at t = 0, equivalently when |S1| = 0.The proof establishes this by showing L(t) decreases over the relevant range and that L(N) < L(0) under the assumption.

B. Imbalanced Case. · B.1. Proofs of Lemma 1 and Proposition 1.

The imbalanced-case analysis reduces the original nonconvex program to a convex program and characterizes its minimizers through that reduction. It then uses the same argument for Proposition 1 and begins studying a partial-classification limit case for Theorem 5.

  • B.1. Proofs of Lemma 1 and Proposition 1.: For any feasible solution of Eq. (7), the constructed matrix X satisfies the constraints of the convex program in Eq. (15).The proof explicitly establishes X ⪰ 0 and verifies the remaining constraints using feasibility of (H, W).
  • B.1. Proofs of Lemma 1 and Proposition 1.: The objective of Eq. (15) lower-bounds the objective of Eq. (7) for every feasible pair (H, W).This comparison is made using the global minimum L0 of Eq. (15).
  • B.1. Proofs of Lemma 1 and Proposition 1.: A minimizer X⋆ of Eq. (15) defines a feasible pair (H⋆, W⋆) for Eq. (7) through Eq. (16).The proof verifies feasibility and uses the associated construction to match the optimization objectives.
  • B.1. Proofs of Lemma 1 and Proposition 1.: Combining Eq. (64) and Eq. (65) shows that L0 is the global minimum of Eq. (7), with (H⋆, W⋆) as a minimizer.The construction includes class-specific vectors k defined from rows of X⋆.
  • B.1. Proofs of Lemma 1 and Proposition 1.: Any minimizer not representable by Eq. (16) would yield unequal within-class features hk′,i and hk′,j for some class k′.The contradiction argument starts from a minimizer X′ of Eq. (15) and identifies i ≠ j with hk′,i ≠ hk′,j.
  • B.1. Proofs of Lemma 1 and Proposition 1.: Under the condition Σk=1^K X⋆(k, k) = EH, all solutions of Eq. (7) have the form specified in Eq. (16).This condition completes the proof’s characterization of all solutions.
  • B.1. Proofs of Lemma 1 and Proposition 1.: Proposition 1 follows by the same argument, so its proof is omitted.The passage explicitly states that the proof is obtained analogously.
  • B. Imbalanced Case.: To prove Theorem 5, the analysis first studies a limit case in which classification is learned for only a partial set of classes.The paper then states that it solves an optimization program for this limit case.

B.2. Proof of Theorem 5.

The proof of Theorem 5 uses Lemma 5 to characterize minimizers of Eq. (66), then derives a contradiction showing the relevant minority-class weights cannot remain separated. Lemma 5 establishes that minority-class features vanish at minimizers and that the global minimum depends only on KA, KB, EH, and EW.

  • Lemma 5: Lemma 5 shows that every minimizer has hk,i = 0p for minority classes k ∈ [KA + 1 : K].This property is used both in characterizing Eq. (66) and in the subsequent proof of Theorem 5.
  • Lemma 5: The global minimum L0 of Eq. (66) depends only on KA, KB, EH, and EW.The same parameter dependence applies to the quantities LA and LB introduced for the decomposed optimization programs.
  • Proof of Theorem 5: Theorem 5 is proved by assuming such a persistent separation, comparing objective values at the minimizers of the decomposed programs, and obtaining a contradiction.The comparison is made between Eq. (70) and Eq. (71).
  • Proof of Lemma 5: For KA = 1, selecting the stated constants makes the inequalities reduce to equalities, establishing the global minimum L1 for Eq. (66).The corresponding construction uses any unit vector u and sets minority-class features to zero.
  • Lemma 5: If two minority-class weight vectors remain separated by a positive distance, Lemma 5 yields a strictly positive objective gap depending on ε, KA, KB, EH, and EW.The gap is independent of the optimization-program index ℓ, enabling the contradiction argument.

PnA

The analysis establishes the global minimum in the considered cases and characterizes its minimizers: minority-class activations must vanish under the model’s bias-free formulation. It also notes that this vanishing does not necessarily mean the corresponding network activations die when biases are included.

  • Global optimality: L2 is the global minimum of Eq. (66) in the analyzed cases, with equality attained by explicitly constructed feasible solutions.The three cases are combined to complete the proof of Lemma 5.
  • Minimizer structure: Any minimizer satisfies hk,i = 0p for all minority classes k ∈ [KA + 1 : K].This condition follows from the equality characterization in Lemma 6 and is repeated in the case analysis.
  • Minimizer structure: The lower-bound equality conditions force minority-class hidden representations to be zero under the Layer-Peeled Model constraints.Lemma 6 derives this equality condition while bounding the objective from below.
  • Interpretation and limitation: The condition hk,i = 0p does not imply that network activations die for minority classes because the analysis omits bias terms.The remark explicitly limits the interpretation of zero hidden representations to the bias-free analysis.

C. Additional Results. · D. Additional Experimental Results.

The additional results justify oversampling’s optimization stability, establish a convex nuclear-norm relaxation for the model, and derive cross-entropy properties of its minimizers. Additional experiments report Minority Collapse accuracies and show ImageNet-pretrained networks do not converge to a Simplex ETF.

  • C. Additional Results.: Oversampling and weight adjusting optimize the same objective but use different optimization algorithms.The comparison treats both as common deep-learning techniques based on Eq. (17).
  • C. Additional Results.: Oversampling has potentially much smaller update variance than weight adjusting, providing a justification for its greater optimization stability.The argument connects update variance to convergence in stochastic optimization.
  • C. Additional Results.: The analysis models weight adjusting by stochastic-gradient updates on randomly sampled data and oversampling by duplicating minority data before stochastic gradient descent.The analysis samples without replacement and considers one datum per update, with extension to mini-batches stated directly.
  • C. Additional Results.: Under matched expected update scales and comparable gradient norms, setting the oversampling ratio wr ≍ nA/nB yields the smaller-variance conclusion.The comparison assumes KA ≍ 1, KB ≍ 1, nA ≫ nB, and 1 ≪ wr ≲ (nA/nB).
  • C. Additional Results.: Program Eq. (7) can be relaxed to a nuclear norm-constrained convex optimization, and an SVD-based construction recovers a minimizer of the original program.Lemma 7 assumes p ≥ K and convexity of the loss in its first argument.
  • D. Additional Experimental Results.: Additional Minority Collapse experiments report corresponding training and test accuracies for the Figure 4 settings.These results are provided in Tables 3–4.
  • D. Additional Experimental Results.: ImageNet-pretrained neural networks officially released by PyTorch do not converge to a Simplex ETF, so neural collapse does not emerge during imbalanced training’s terminal phase.ImageNet is described as an imbalanced dataset with K = 1000 classes.
Loading 2101.12699v3…