PaperlayerView exact revision on arXiv

Source-linked AI summary

Long-tailed Visual Recognition via Gaussian Clouded Logit Adjustment

Mengke Li, Yiu-ming Cheung, Yang Lu

arXiv:2305.11733v1cs.CV

TL;DR

Long-tailed training compresses tail-class embeddings and biases classification toward head classes, while softmax saturation weakens training gradients. The paper proposes Gaussian clouded logit adjustment with larger tail-class cloud sizes, together with CBEN classifier re-training, and reports superior benchmark performance. Its Gaussian feature assumption constrains the embedding calibration formulation.

  • Problem

    Long-tailed training produces distorted embedding spaces and classifiers biased toward head classes, while softmax saturation weakens sample validity through early gradient vanishing.

  • Method

    GCL perturbs class logits with Gaussian noise of varied amplitudes, assigning larger cloud sizes to tail classes, while CBEN re-balances classifier training using sample validity and label frequencies.

  • Results

    The proposed method surpasses recently proposed counterparts across multiple long-tailed benchmark datasets.

  • Takeaways & Limitations

    Using softmax saturation to vary sample validity and CBEN to balance the classifier yields an evenly distributed embedding space and improved GCL performance.

  • Takeaways & Limitations

    The embedding calibration assumes features of different class samples follow Gaussian distributions, and the disturbance parameter should remain small because large disturbances can mislead the model.

Abstract

from arXiv · show

Long-tailed data is still a big challenge for deep neural networks, even though they have achieved great success on balanced data. We observe that vanilla training on long-tailed data with cross-entropy loss makes the instance-rich head classes severely squeeze the spatial distribution of the tail classes, which leads to difficulty in classifying tail class samples. Furthermore, the original cross-entropy loss can only propagate gradient short-lively because the gradient in softmax form rapidly approaches zero as the logit difference increases. This phenomenon is called softmax saturation. It is unfavorable for training on balanced data, but can be utilized to adjust the validity of the samples in long-tailed data, thereby solving the distorted embedding space of long-tailed problems. To this end, this paper proposes the Gaussian clouded logit adjustment by Gaussian perturbation of different class logits with varied amplitude. We define the amplitude of perturbation as cloud size and set relatively large cloud sizes to tail classes. The large cloud size can reduce the softmax saturation and thereby making tail class samples more active as well as enlarging the embedding space. To alleviate the bias in a classifier, we therefore propose the class-based effective number sampling strategy with classifier re-training. Extensive experiments on benchmark datasets validate the superior performance of the proposed method. Source code is available at https://github.com/Keke921/GCLLoss.

1. Introduction

Long-tailed training distorts the embedding space and biases the classifier toward head classes. The paper proposes Gaussian clouded logit adjustment and class-based effective number sampling to address these problems, with experiments showing superior performance.

  • Motivation: Long-tailed data compresses tail-class spatial spans and biases classifier learning toward head classes.The paper identifies distorted embedding space and biased classifier as the two key problems of direct training.
  • Motivation: Post-hoc calibration can adjust class boundaries but cannot correct the distorted embedding space.
  • Gaussian Clouded Logit Adjustment: GCL perturbs class logits with Gaussian noise of different amplitudes, assigning larger cloud sizes to tail classes.This reduces softmax saturation and gives tail samples more active gradients without overfitting.
  • Classifier Re-training: CBEN sampling re-balances the classifier using sample validity and label frequencies, avoiding excessive tail-class training.The strategy is designed to mitigate classifier bias and further improve GCL.
  • Results: Extensive experiments on multiple long-tailed benchmark datasets show that GCL surpasses recently proposed counterparts.

2. Related Works

Related work addresses long-tailed recognition through loss modification, logit adjustment, and representation-classifier decoupling. These approaches respectively alter training emphasis, class margins, or the stages used to learn representations and classifiers.

  • Overview: Long-tailed recognition methods are grouped into loss modification, logit adjustment, and representation-classifier decoupling.
  • Loss Modification: Re-weighting methods modify loss coefficients to emphasize difficult samples or compensate for class imbalance.Focal loss uses a tunable focusing parameter, while class-wise methods assign class-dependent weights.
  • Logit Adjustment: Logit adjustment assigns relatively large margins to tail classes using label-frequency information or test-set distributions.
  • Representation-Classifer Decoupling: Decoupling methods learn representations first and re-balance or fine-tune the classifier in a second stage.The literature includes deferred re-weighting, class-balanced resampling, and adaptive calibration strategies.

3. Proposed Approach: GCL

The proposed GCL approach perturbs class logits with Gaussian noise to exploit softmax saturation, using larger cloud sizes for tail classes to calibrate the embedding space. It then applies the resulting clouded logits in a modified softmax loss.

  • Motivation: Softmax saturation makes target-class gradients rapidly approach zero as logit differences increase, worsening overlap and tail-class embedding compression.The paper uses this limitation as motivation for recalibrating the embedding space.
  • Gaussian Clouded Logits: GCL perturbs features with Gaussian noise before computing classifier logits, producing randomly varied Gaussian clouded logits.The disturbance is applied to the embedding feature and propagated through the classifier.
  • Gaussian Clouded Logits: Clouded cosine logits normalize disturbed features and class anchors, with class-specific cloud sizes controlling the perturbation magnitude.The normalized formulation represents the disturbance through the class cloud size and Gaussian noise magnitude.
  • Cloud Size Design: Tail classes receive larger cloud sizes because their training samples are fewer, while the most frequent class receives cloud size zero.The proposed setting is δ_j = log n_max − log n_j.
  • Cloud Size Design: Negative Gaussian perturbations reduce tail-class logit differences, alleviating softmax saturation and enlarging tail-class decision margins.The paper argues that correctly predicting a feature farther from its class anchor supports classification of closer features.
  • GCL Loss: The GCL loss inserts clouded cosine logits into the original softmax objective to train the model with class-dependent perturbations.The loss averages the negative log probability of each target class over the training samples.

7 end

The training procedure iterates over batches sampled from the original long-tailed data and calculates the loss using the prescribed equation.

  • Training Procedure: Each iteration samples a batch from the original long-tailed dataset before calculating the loss by Eq. (12).The listed procedure includes batch sampling followed by loss computation.

13 end

The CBEN strategy assigns class-dependent sampling probabilities based on effective sample validity and normalizes them across the dataset.

  • Sampling Probability: Sampling probabilities are normalized so that their sum over all data equals one.The normalization is applied after calculating the class-specific probabilities.
  • Effective Number: β_j reflects class-sample validity and increases with cloud size, so classes with larger clouds participate more in training.The paper links β_j positively to the class cloud size δ_j.
  • Effective Number: β_j is linearly mapped from the cloud-size range into [a, a + b], where a and b are range hyper-parameters.The mapping uses the minimum and maximum cloud sizes for normalization.

4. Experiments

The method is evaluated on five long-tailed benchmark settings spanning CIFAR, ImageNet, iNaturalist, and Places datasets with varied class counts and imbalance levels.

  • Benchmarks: Experiments use CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, iNaturalist 2018, and Places-LT.The benchmarks include both constructed long-tailed versions and naturally imbalanced data.
  • CIFAR-LT: CIFAR-10-LT and CIFAR-100-LT contain 10 and 100 classes, respectively, with imbalance ratios γ of 50, 100, and 200.The long-tailed training sets are created by exponential class-wise down-sampling while validation sets remain unchanged.
  • ImageNet-LT and Places-LT: ImageNet-LT contains 115.8K training images from 1,000 categories and is constructed by Pareto-distribution truncation.The cited passage reports an imbalance ratio of γ = 1,280/5.
  • iNaturalist 2018: iNaturalist 2018 contains 437.5K training images and 24.4K validation images across 8,142 categories.The experiments follow the official training and validation splits.

4.2. Experimental Setting

The experiments use Gaussian perturbation settings with bounded amplitudes and standard optimization protocols across long-tailed datasets and model scales.

  • The Gaussian parameters are set to μ = 0 and σ = 1/3, with perturbations clamped to [−1, 1].The feature cloud size is bounded by 1, and sample validity β_j is set within [0.999, 0.9999].
  • The implementation uses PyTorch backbones.
  • Models use SGD with momentum 0.9 and a multi-step learning-rate schedule.Most models are trained from scratch; ResNet-152 is initialized from balanced ImageNet-2012 pre-training.
  • ResNet-32 is used for CIFAR-10/100-LT, while large-scale datasets follow prior long-tailed training settings.The second stage re-balances classifiers following the cited decoupling protocol.

4.3. Competing Methods

The study compares GCL against vanilla, mixup-enhanced, representation-learning, logit-adjustment, and two-stage long-tailed recognition methods.

  • Baseline Methods: Vanilla cross-entropy and cross-entropy with mixup serve as baseline methods.Mixup is included because prior visual-recognition work has reported its efficacy.
  • State-of-the-art Methods: The comparisons include OLTR and De-confound-TDE inference as representation-learning and logit-adjustment methods.
  • State-of-the-art Methods: Two-stage comparisons include LDAM-DRW and MisLAS, with BBN and contrastive learning additionally evaluated on CIFAR datasets.
  • State-of-the-art Methods: Large-scale datasets are also compared against decoupling and logit-adjustment approaches.

4.4. Comparison Results

GCL outperforms prior methods across CIFAR-10/100-LT and three large-scale long-tailed benchmarks under top-1 accuracy evaluation.

  • Top-1 accuracy is used to evaluate the comparative results.Results from studies without released code or hyperparameters are quoted from their original papers.
  • CIFAR-10/100-LT: GCL outperforms previous methods across all evaluated imbalance ratios on CIFAR-10/100-LT.The largest improvement is reported at γ = 200.
  • CIFAR-10/100-LT: 79.03% and 44.88% top-1 accuracy are achieved on CIFAR-10-LT and CIFAR-100-LT at γ = 200.These results exceed MisLAS by 1.72% and 2.55%, respectively.
  • Large-scale datasets: 54.88% top-1 accuracy on ImageNet-LT exceeds DisAlign by 1.97% and MisLAS by 2.77%.
  • Large-scale datasets: 72.01% top-1 accuracy on iNaturalist 2018 exceeds the second-best method by 0.44%.On Places-LT, GCL reaches 40.64%, gaining 0.49% over MisLAS.

4.5. Model Validation and Analysis

Ablations and embedding visualizations indicate that GCL produces more separable embeddings, while CBEN sampling and cRT improve classifier re-balancing.

  • The Role of Gaussian Clouded Logit: GCL embeddings are more scattered across classes than cross-entropy embeddings in the CIFAR-10-LT t-SNE visualization.The test-set visualization reports a smaller obscure region for GCL.
  • The Role of Gaussian Clouded Logit: The cloud-size ablation compares cosine, power-difference, and logarithmic-difference adjustment strategies.The logarithmic-difference strategy is selected based on the reported ablation.
  • Classifier Re-balance Strategies: CBEN is compared with instance balance, class balance, and effective-number sampling under cRT.
  • Classifier Re-balance Strategies: cRT improves top-1 accuracy by 1.64% and performs best among the evaluated classifier re-training strategies.The comparison uses CIFAR-10-LT with γ = 100.
  • Classifier Re-balance Strategies: GCL without classifier re-training surpasses BBN by 0.7% in the reported comparison.The authors also report that IB+cRT degrades performance, indicating possible classifier overfitting.

5. Conclusion

The proposed method uses softmax saturation to adjust class-specific sample validity, then combines GCL with CBEN sampling and cRT to address embedding distortion and classifier bias.

  • GCL assigns relatively large cloud sizes to tail-class logits, encouraging tail samples to participate in training and enlarging embedding margins.
  • GCL changes the effectiveness of different classes by exploiting softmax saturation’s unequal effects on head and tail classes.
  • CBEN sampling with cRT rebalances the classifier after GCL and further boosts model performance.
  • Experiments on various benchmark datasets demonstrate superior performance compared with existing state-of-the-art methods.
Loading 2305.11733v1…