PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Representation Geometry in Supervised Contrastive Learning under Long-Tailed Data

Joonhyeok Jeong, Jy-yong Sohn, Kibok Lee

arXiv:2610.32039v1cs.LGcs.AIcs.CVstat.ML

TL;DR

Under long-tailed data, whether supervised contrastive objectives form a regular simplex at their optimum remains insufficiently understood. This paper develops a framework to test that property, finds SCL and BCL do not guarantee it, and introduces structural balancing and FBCL, which has a regular-simplex optimum and improves recognition and transfer results.

  • Problem

    Whether supervised contrastive objectives form a regular simplex at their optimum under long-tailed distributions remains insufficiently understood.

    Source: §1 · p. 1

  • Method

    The paper derives class-level lower bounds to test simplex optimality, then applies structural balancing to the anchor aggregation and proposes FBCL.

    Source: §1 · pp. 1–2

  • Results

    FBCL admits a regular-simplex global optimum, and experiments report improvements in long-tailed recognition and representation transfer.

    Source: §7 · pp. 9–10

  • Takeaways & Limitations

    The results provide a framework for analyzing and improving supervised contrastive learning under long-tailed distributions.

    Source: §7 · p. 10

    The theory uses an all-classes batch setting, and its dimensional condition is satisfied on ImageNet-LT but not iNaturalist.

    Source: §3 · p. 3, §6 · p. 9

Abstract

from arXiv · show

Supervised contrastive learning and its balanced variants have demonstrated their effectiveness for representation learning under long-tailed distributions. However, although prior analyses suggest that their optimal representations may exhibit a regular simplex geometry, whether such a geometry is indeed exhibited remains insufficiently understood. In this work, we present a theoretical framework for these objectives that characterizes whether a regular simplex is formed at the optimum. By applying this to representative objectives, we show that these objectives fail to form a regular simplex at the optimum because they remain dependent on the number of samples per class. Motivated by this analysis, we introduce structural balancing that can be applied to contrastive objectives, and propose a fully balanced contrastive objective that provably forms a regular simplex geometry at its optimum. Both theoretical and empirical results demonstrate that the proposed method yields a more balanced representation space and improves performance on long-tailed recognition and transfer learning tasks.

1 INTRODUCTION

Long-tailed data can skew learned representations, while the optimal geometry of supervised contrastive objectives under imbalance remains insufficiently understood. The paper analyzes regular-simplex optimality and introduces structural balancing and FBCL to address class-count dependence.

1 INTRODUCTION

  • SCL and BCL do not have a regular simplex as a global minimizer under long-tailed class imbalance.Their losses retain dependence on the number of samples per class.Source: §1 · pp. 1–2
  • Under balanced sampling, SCL, BCL, and FBCL all form a regular simplex; at a 1000:10:1 imbalance, only FBCL does.Figure 1 visualizes class-center angles for three sampled ImageNet classes.Source: Figure 1 · p. 2
  • Structural balancing equalizes anchor contributions in the overall objective, removing the remaining class-count dependence.It extends balancing beyond the individual anchor-wise losses used by SCL and BCL.Source: §1 · pp. 1–2
  • FBCL is a contrastive objective whose global minimizer forms a regular simplex.The paper presents it as an objective built using structural balancing.Source: §1 · p. 2
  • Structural balancing makes the intra-class variance penalty approximately independent of class frequency, and experiments report improvements in recognition and transfer.The paper also reports that structural balancing combines effectively with other methods.Source: §1 · p. 2

2 RELATED WORK

Prior long-tailed recognition work includes data and loss balancing, specialized architectures, logit adjustment, and transfer from head classes. In contrast, this paper analyzes regular-simplex optimality for supervised contrastive objectives and proposes a balancing method with that geometry as its optimum.

Long-Tailed Recognition

  • Earlier long-tailed approaches include re-sampling, re-weighting, two-branch or two-stage designs, logit adjustment, and transfer methods.Foundation-model fine-tuning has also been studied, though the impact of imbalanced data on it is not fully understood.Source: §2 · p. 2

Contrastive Learning under Long-Tailed Distributions

  • Long-tailed contrastive methods include PaCo, KCL, BCL, and TSC, which use different mechanisms to address class imbalance.Reported mechanisms include learnable class centers, equal positive-sample counts, balanced negative terms, and uniform class targets.Source: §2 · p. 3
  • This paper provides a framework for testing regular-simplex optimality and shows that neither SCL nor BCL has that geometry as a global minimizer.The result is nontrivial for BCL, previously shown to admit such geometry at its minimum.Source: §2 · p. 3
  • The paper proposes structural balancing and a fully balanced contrastive loss whose global minimizer is a regular simplex under long-tailed distributions.It applies its framework to SCL and BCL.Source: §2 · p. 3

3 PRELIMINARIES

The analysis considers unit-sphere representations and long-tailed class counts, using an all-classes batch as a theoretical setting. It treats a regular simplex as a target geometry associated with class collapse and separated class centers.

3 PRELIMINARIES

  • The theoretical setting assumes a batch containing all K classes, with B > 4, K ≥ 3, and class counts ordered from head to tail with B1 ≫ BK.Practical stochastic minibatch experiments instead use the class-complement setting, which adds class centers as contrastive pairs.Source: §3 · p. 3
  • The encoder maps inputs to unit-norm representations on a hypersphere, and the analysis defines SCL and BCL losses over features and labels.The temperature parameter is set to τ = 1 for simplicity.Source: §3 · pp. 3–4
  • A regular simplex has pairwise class-center inner products of −1/(K−1), subject to K ≤ h + 1.The paper treats it as an optimal target geometry combining within-class collapse and between-class separation.Source: §3 · p. 4

4 REPRESENTATION GEOMETRY UNDER LONG-TAILED DISTRIBUTIONS

The paper analyzes supervised contrastive representation geometry through class-level lower bounds, shows why SCL and BCL fail to guarantee a regular simplex under imbalance, and introduces structural balancing and FBCL. It also examines how structural balancing changes the intra-class variance penalty.

4.1 GEOMETRIC ANALYSIS OF SUPERVISED CONTRASTIVE OBJECTIVES

  • The class-level lower bound reduces feature-space analysis to class centers, with equality when features within each class collapse to its center.A center configuration that is not optimal for the lower bound cannot be optimal for the original loss.Source: §4.1 · pp. 4–5
  • The framework derives a condition for a regular simplex to minimize the lower bound, then transfers lower-bound non-optimality to the original objective.It is applied to representative supervised contrastive objectives.Source: §4.1 · p. 5
  • Neither SCL nor BCL has a regular simplex as a global minimizer under class imbalance.For BCL, anchor-wise balancing of negative terms still leaves class-count dependence in the lower bound.Source: §4.1 · p. 5

4.2 FULLY BALANCED CONTRASTIVE LEARNING VIA STRUCTURAL BALANCING

  • SCL and BCL balance contributions within each anchor-wise loss, but summing those losses weights classes by their cardinality.Structural balancing instead balances anchor contributions across classes in the overall objective.Source: §4.2 · pp. 5–6
  • Structural balancing removes class-count dependence from the FBCL lower bound by assigning every class weight 1/K.The paper then proves that the regular-simplex centers and their collapsed feature configuration minimize the FBCL objectives.Source: §4.2 · p. 6
  • FBCL has a global minimum if and only if representations collapse to class centers forming a regular simplex.Its class-level lower bound assigns equal weight 1/K to each class and removes class-cardinality coefficients.Source: §4.2 · p. 6

4.3 ANALYSIS OF INTRA-CLASS VARIANCE PENALTY

  • Under SCL and BCL, the intra-class variance penalty is approximately proportional to class cardinality, while structural balancing makes it approximately independent of cardinality.The paper interprets the latter as a nearly uniform variance penalty across classes.Source: §4.3 · p. 7

5 EXPERIMENTS

The experiments evaluate long-tailed recognition and transferability using a dual-branch framework, then compare contrastive objectives and balancing components. FBCL achieves the highest average transfer performance and the highest results among contrastive methods based on SCL or BCL, while ablations find PB, NB, and SB complementary.

5.1 DATASETS

  • Experiments assess classification on ImageNet-LT and iNaturalist, plus transferability across diverse downstream datasets under linear evaluation and fine-tuning.ImageNet-LT and iNaturalist results are reported by many-, medium-, few-shot, and all-class accuracy.Source: §5 · p. 7, §5.1 · p. 7

5.2 DUAL-BRANCH TRAINING FRAMEWORK

  • The dual-branch model uses a base encoder, prediction head, and contrastive head, optimizing logit-adjusted cross-entropy and a contrastive loss together.The overall objective weights the two losses with hyperparameters λ and µ.Source: §5.2 · p. 7

5.3 IMPLEMENTATION DETAILS

  • The implementation follows Zhu et al. (2022), uses ResNet-50 and RandAug, and compares SimCLR, SCL, and BCL within the same training framework.The contrastive-head dimension is 1024 for ImageNet-LT and iNaturalist and 128 for CIFAR-100-LT; transfer evaluation follows ImageNet-LT pretraining.Source: §5.3 · pp. 7–8

5.4 RESULTS

  • FBCL achieves the highest average performance on both transfer evaluations and the highest results among contrastive methods based on SCL or BCL.Table 3 compares transferability across 11 natural-image classification datasets; Table 2 compares long-tailed recognition methods on ImageNet-LT and iNaturalist.Source: §5.4 · pp. 8–9, Table 3 · p. 8, Table 2 · p. 8

5.5 ABLATION STUDY

  • On ImageNet-LT, ablation results show PB, NB, and SB are complementary: SB reduces disparity in intra-class variance across classes, while PB reduces variance across all groups and improves overall accuracy.Without SB, gaps are 0.168 for SCL and 0.195 for BCL; adding SB yields 0.141 for PB+SB and 0.143 for FBCL, and removing NB decreases accuracy.Source: §5.5 · p. 9

6 DISCUSSION

The theoretical analysis assumes batches contain all classes and characterizes the objective's geometry at its optimum under that setting. Whether the geometric optima are attainable at large scale or under partial class coverage remains underexplored.

6 DISCUSSION

  • The theory uses all-class batches; attainability at large scale and geometry under partial class coverage or K > h+1 remain open questions.The dimensional condition holds on ImageNet-LT but not iNaturalist, while structural balancing still improves combined methods in practical settings including iNaturalist.Source: §6 · p. 9

7 CONCLUSION

The paper develops a framework for testing regular-simplex optimality and finds that SCL and BCL do not attain that geometry under class imbalance. Structural balancing removes the class-count dependence, and FBCL has a regular simplex as a global optimum.

7 CONCLUSION

  • FBCL admits a regular simplex as a global optimum, while SCL and BCL do not under class imbalance.The conclusion attributes the failure to class-cardinality dependence in the aggregation of anchor-wise losses.Source: §7 · p. 9
  • Experiments show improvements in long-tailed recognition and representation transfer, and effectiveness when structural balancing is combined with other methods.The conclusion names ImageNet-LT, iNaturalist, and transfer tasks.Source: §7 · pp. 9–10

B THEORETICAL FRAMEWORK AND PROOFS

  • The framework derives a class-level lower bound and uses its optimality conditions to assess regular-simplex geometry.A configuration that fails to minimize the lower bound cannot minimize the original loss; the bound's equality condition corresponds to class collapse.Source: B THEORETICAL FRAMEWORK AND PROOFS · p. 14

B.1 REDUCTION FROM FEATURE-LEVEL TO CLASS-LEVEL

  • Class-collapsed features are global minimizers of the original loss when their centers minimize the class-level lower bound.The framework also establishes that a center configuration that does not minimize the lower bound cannot minimize the original loss.Source: §B.1 · pp. 14, 16

B.2 A CONDITION OF THE LOWER BOUND FOR A REGULAR SIMPLEX OPTIMALITY

  • A necessary condition for a regular simplex to minimize the lower bound is expressed through the objective's coefficients.Those coefficients arise during the lower-bound derivation.Source: §B.2 · pp. 17–18

B.3 APPLYING OUR FRAMEWORK TO SUPERVISED CONTRASTIVE OBJECTIVES

  • SCL and BCL retain dependence on class cardinalities, whereas structural balancing removes that dependence for FBCL.For FBCL, the condition for regular-simplex optimality holds, and further lower bounds establish that this stationary point is a global minimum.Source: §B.3 · p. 18

B.3.1 SCL DOES NOT YIELD A REGULAR SIMPLEX AT THE OPTIMUM UNDER LONG-TAILED DISTRIBUTIONS

Under long-tailed distributions, SCL does not attain a regular simplex as the geometry of its global minimum. The proof shows that class-cardinality dependence prevents the simplex from satisfying the lower bound's first-order condition.

B.3.1 SCL DOES NOT YIELD A REGULAR SIMPLEX AT THE OPTIMUM UNDER LONG-TAILED DISTRIBUTIONS

  • Under long-tailed distributions, SCL does not attain a regular simplex as the geometry of its global minimum.The proof establishes that the simplex fails the lower bound's first-order Lagrangian condition, so it is not a global minimizer.Source: §B.3.1 · pp. 18–19
  • Class-cardinality dependence prevents the regular simplex from satisfying the first-order condition for the SCL lower bound.The proof uses classes with unequal sample counts to show that the condition cannot hold.Source: §B.3.1 · p. 19
  • The cited prior analyses did not show SCL's failure to form a regular simplex without additional assumptions.The paper also notes that those works did not propose an objective that provably induces a regular simplex under class imbalance.Source: §B.3.1 · p. 19

B.3.2 BCL DOES NOT YIELD A REGULAR SIMPLEX AT THE OPTIMUM UNDER LONG-TAILED DISTRIBUTIONS

The analysis finds that BCL does not attain a regular simplex at its optimum, while FBCL does. It derives these results by bounding anchor-wise and class-wise losses and examining the resulting class-center geometry.

B.3.2 BCL DOES NOT YIELD A REGULAR SIMPLEX AT THE OPTIMUM UNDER LONG-TAILED DISTRIBUTIONS

  • BCL’s final lower bound is not tight because Jensen’s inequality is applied over classes in one step; the regular-simplex centers therefore fail the Lagrangian condition and are not optimal.The regular-simplex configuration fails to minimize the class-level lower bound.Source: §B.3.2 · p. 20

B.3.3 FBCL YIELDS A REGULAR SIMPLEX AT THE OPTIMUM UNDER LONG-TAILED DISTRIBUTIONS

  • FBCL’s regular-simplex configuration satisfies the Lagrangian condition and attains the lower bound, making it a global minimizer.The lower bound is attained if and only if the configuration is a regular simplex.Source: §B.3.3 · pp. 21–22

B.4 LOWER BOUNDS OF SUPERVISED CONTRASTIVE OBJECTIVES

  • The three class-level lower bounds share a two-step derivation—first bounding anchor-wise losses, then class-wise losses—and differ in negative-class weights and anchor-loss aggregation.FBCL additionally modifies the aggregation of anchor-wise losses.Source: §B.4 · p. 22

B.4.1 LOWER BOUND OF THE ANCHOR-WISE LOSS (LEMMA 3)

  • Lemma 3 bounds each anchor-wise loss, with equality requiring equal anchor similarities among samples within each class.The equality conditions include constant similarities to positive samples and to samples of each negative class.Source: §B.4.1 · p. 23

B.4.2 LOWER BOUND OF THE CLASS-WISE LOSS (LEMMA 4)

  • Lemma 4 bounds the class-wise loss, with equality requiring each class’s anchors to have the same similarity to each negative-class center.Source: §B.4.2 · p. 24

B.4.3 PROOFS OF PROPOSITIONS 1, 2, AND 3

The propositions derive lower bounds for SCL, BCL, and FBCL, with equality at class collapse. The section also describes loss decomposition, FBCL’s balancing components, and the toy experiment setup.

B.4.3 PROOFS OF PROPOSITIONS 1, 2, AND 3

  • For SCL, BCL, and FBCL, the sequential anchor-wise and class-wise bounds have equality when features collapse to their class centers.This class-collapse condition alone does not specify the geometry of the class centers.Source: §B.4.3 · pp. 24–25

B.5 DECOMPOSITION OF LOSSES

  • The loss decomposition separates alignment and uniformity, but uniformity cannot be feasibly expressed in terms of intra-class variance without assumptions.Source: §B.5 · p. 26

B.6 BALANCING COMPONENTS IN FBCL

  • FBCL combines positive balancing over positive samples, negative balancing over negative samples per class, and structural balancing across anchor-wise losses.Without positive balancing, the positive-sample summation corresponds to SimCLR.Source: §B.6 · p. 28

C.1 FIGURE 1 DETAILS

  • The toy visualization uses three random ImageNet classes resized to 32 × 32 and a two-layer MLP that maps images to two-dimensional normalized embeddings.The embeddings lie on the unit circle in the two-dimensional feature space.Source: §C.1 · p. 28

C.2 IMPLEMENTATION DETAILS AND HYPERPARAMETERS

These implementation details distinguish the theory’s full-class anchor coverage and fixed temperature from the mini-batch practice, where class coverage depends on batch composition.

C.2 IMPLEMENTATION DETAILS AND HYPERPARAMETERS

  • Theory assumes all classes appear as anchors in each batch and fixes τ=1, while practical temperature follows baseline settings and affects representations under balanced and imbalanced distributions.Source: §C.2 · p. 28
  • ImageNet-LT training lasts 90 epochs and iNaturalist training 100 epochs; both use SGD with momentum 0.9 and batch size 256.Source: §C.2 · pp. 28–29
  • The reported setup uses batch size 256 and τ=0.07, with class-complement pairs except for SCL and SimCLR; CIFAR-100-LT instead uses τ=0.1.Source: §C.2 · p. 29

C.3 TRANSFER LEARNING EVALUATION PROTOCOL

The appendix describes transfer-learning evaluation protocols and reports the scope of toy-experiment and dataset statistics tables. The supplied passages do not include the table values.

C.3 TRANSFER LEARNING EVALUATION PROTOCOL

  • Linear evaluation tunes L2-regularized cross-entropy with L-BFGS on a validation split, then retrains using training and validation data before test evaluation; fine-tuning selects hyperparameters on validation data.The transfer-learning section also identifies Table 5 as dataset information.Source: §C.3 · p. 29, Table 5 · p. 29

C.4 TOY EXPERIMENT

  • The toy experiment uses five Gaussian classes with cardinalities 500, 250, 100, 50, and 25, and reports average normalized intra-class variance by class group.The test set has 500 samples, with 100 per class.Source: §C.4 · pp. 29–30

C.5.1 FBCL ON IMAGENET-LT AND INATURALIST

C.5.2 INTRA-CLASS VARIANCE AND ACCURACY

  • The appendix identifies five-run tables for synthetic-data intra-class variance and Top-1 accuracy, and balancing-component ablations on CIFAR-100-LT; it also lists a three-run ImageNet-LT balancing ablation.Table 8 concerns CIFAR-100-LT with imbalance ratio 100.Source: §C.5.2 · p. 30, Table 7 · p. 30, Table 8 · p. 30

C.5.3 STRUCTURAL BALANCING WITH OTHER METHODS

Structural balancing outperforms the compared sampling and reweighting strategies, while FBCL produces more separated class-center geometry than BCL. The appendix also reports structural-balancing combinations with other methods, including improved few-class accuracy for ProCo on CIFAR-100-LT.

C.5.3 STRUCTURAL BALANCING WITH OTHER METHODS

  • Structural-balancing combinations include FBCL and BCL with DBM, ProCo on ImageNet-LT and iNaturalist, and other methods on CIFAR-100-LT; ProCo gains few-class accuracy on CIFAR-100-LT.The official Wang et al. (2024) implementation lacked ImageNet-LT and iNaturalist code, so those results were deferred.Source: Table 13 · p. 31, Table 10 · p. 31, Table 11 · p. 31

C.6 COSINE SIMILARITY OF THE REPRESENTATION SPACE

  • On CIFAR-100-LT and ImageNet-LT, FBCL has lower class-center cosine similarity than BCL, especially for few-shot classes, indicating more separated inter-class geometry.The CIFAR-100-LT comparison spans imbalance ratios and classes; ImageNet-LT shows the same trend across shot groups and overall.Source: §C.6 · pp. 31–32, Table 15 · p. 32, Table 14 · p. 32

C.7 CLASS BALANCED SAMPLING AND OTHER REWEIGHTING STRATEGIES

  • At IR=200, BCL+re-weighting reaches 46.35 ± 0.34 overall accuracy versus BCL (47.63 ± 0.30) and FBCL (48.18 ± 0.32); class-balanced sampling has the lowest overall accuracy.CB-weighting also performs worse than conventional re-weighting, while structural balancing achieves the best performance in the comparison.Source: Table 16 · p. 32, §C.7 · pp. 32–33
  • Structural balancing removes class-frequency dependence in the lower bound and recovers optimal representation geometry, whereas mini-batch reweighting can retain frequency dependence.The comparison notes that full-batch reweighting and structural balancing coincide, but full-batch training is impractical for large-scale datasets.Source: §C.7 · pp. 32–33

Related papers

Loading 2610.32039v1…