Self Supervision to Distillation for Long-Tailed Visual Recognition

• Class re-balancing can overfit tail classes, while decoupled training addresses overfitting but still handles label imbalance poorly during representation learning.Re-balancing methods include resampling and cost-sensitive reweighting.Source: §1 · p. 1
• Soft labels capture relations between classes and can transfer knowledge from head classes to tail classes within multi-stage training.The paper presents soft labels as a way to incorporate label correlation into the training strategy.Source: §1 · p. 1
• SSD combines a self-distillation framework that mines label relations with a self-supervision-guided module that generates less biased, more informative soft labels.Self-supervision uses the long-tailed training set and treats images equally when generating soft labels.Source: §1 · p. 2
• The training pipeline learns an initial teacher with label and self-supervision, adjusts a separate classifier to produce soft labels, then trains a self-distillation network with soft and hard labels.Two classification heads handle the two supervision types because of their semantic gap and possible head-class bias.Source: §1 · p. 2
• SSD leverages soft-label modeling in feature learning to combine re-balanced sampling with decoupled training.Source: §1 · p. 2
• SSD achieves state-of-the-art performance on ImageNet-LT, CIFAR100-LT, and iNaturalist 2018.Source: §1 · p. 2

2.1. Long-tailed Classification

• Re-balancing methods resample data or alter losses, but may overfit data-scarce classes or underfit data-abundant classes; decoupling transfers frozen head-class features during classifier training.Other approaches augment tail classes, adjust class weights or margins, group classes, or modify negative gradients.Source: §2.1 · pp. 2–3

2.2. Learning with Distilled Labels

• Unlike BAN and self-training, SSD produces distilled labels through an additional classifier-adjustment stage for long-tailed data.LFME distills experts trained on category groups, while RIDE distills between models with different numbers of experts; SSD uses distilled labels in multi-stage training to transfer knowledge from head to tail classes.Source: §2.2 · p. 3

2.3. Self-supervised Learning

• Self-supervised learning uses proxy tasks without manual annotations; SSD uses an auxiliary self-supervised task to improve distilled-label quality, unlike self-supervised pretraining in.Examples of proxy tasks include predicting image context or rotation, colorization, jigsaw puzzles, and instance discrimination.Source: §2.3 · p. 3

3.1. Overall framework

• SSD first learns features with supervised and self-supervised tasks, then generates intermediate soft labels, and finally trains with self-distillation.The stages use instance-balanced sampling, class-balanced sampling with the feature backbone fixed, and hybrid supervision, respectively.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2. Feature learning enhanced by self-supervision

• Balanced self-supervised tasks treat images equally without label influence, complementing semantic classification under the original long-tailed distribution.The paper uses rotation prediction and instance discrimination as proxy tasks.Source: §3.1 · p. 3, §3.2 · p. 4

3.1. Overall framework

• A classifier tuned with class-balanced sampling on frozen features produces soft labels that integrate label-driven and data-driven information.The method uses Learnable Weight Scaling (LWS) to rescale classifier weights and reduce the tendency toward head classes.Source: §3.1 · pp. 3–4

3.3. Long-tailed recognition via self-distillation

• The final network is trained from scratch with separate hard-label and soft-label classifiers, combining cross-entropy and distillation losses.The paper states that this self-distilled model can outperform the teacher; classifier fine-tuning is optional but recommended for further improvement.Source: §3.1 · p. 4, §3.3 · p. 5

4.1. Experimental settings

• ImageNet-LT is sampled from ImageNet-2012 with 1,000 classes and 115.8K training images, ranging from 1,280 to 5 images per class.Its validation and test sets are balanced and contain 20K and 50K images, respectively; hyperparameters are selected on validation data.Source: §4.1 · p. 5
• CIFAR100-LT uses 100 categories and imbalance factors of 10, 50, and 100, defined as the ratio of the most- to least-frequent class counts.The validation set contains 100 images per class.Source: §4.1 · p. 5
• iNaturalist 2018 contains 8,142 fine-grained species, 437.5K training images, and an imbalance factor of 500.The official validation set, with 3 images per class, is used for testing.Source: §4.1 · p. 5
• Evaluation reports top-1 accuracy on the corresponding balanced validation or test set, plus Many-shot, Medium-shot, and Few-shot accuracy for ImageNet-LT.The ImageNet-LT groups are defined by at least 100, 20–100, and at most 20 training samples per class, respectively.Source: §4.1 · p. 5

4.2.1 Experimental results on ImageNet-LT

• 56.0% overall accuracy sets a new ImageNet-LT state of the art; SSD beats CIFAR100-LT LWS by 3.7%–4.5% and iNaturalist LWS by 2.7% even with its longer scheduler.The CIFAR100-LT gains span imbalance factors 100, 50, and 10; the iNaturalist comparison is against the 69.5% longer-scheduler LWS baseline.Source: §4.2.1 · p. 6, §4.2.2 · p. 6, §4.2.3 · p. 7
• ImageNet-LT comparisons use ResNeXt-50, and the authors report consistent gains over re-implemented cRT and LWS baselines on every split.They also report that smaller batch size and learning rate improve performance (50.7% v.s. 49.9%), while the 1.5× scheduler adds +1.4% for LWS.Source: §4.2.1 · pp. 5–6, Table 1 · p. 6

4.2.2 Experimental results on CIFAR100-LT

• SSD achieves state-of-the-art performance on CIFAR100-LT across imbalance factors 100, 50, and 10, beating LWS by 3.7%–4.5%.The experiments use ResNet-32; phase I uses rotation prediction as its self-supervised task.Source: §4.2.2 · p. 6, Table 2 · p. 6

4.2.3 Experimental results on iNaturalist 2018

• On iNaturalist 2018, SSD improves over the 69.5% longer-scheduler LWS baseline by 2.7%; it also gains 2.7% over LWS with the 1× scheduler.The comparison uses ResNet-50 and evaluates 1× and 2× training schedulers.Source: Table 3 · p. 6, §4.2.3 · p. 7

4.3.1 Effectiveness of each training stage

• Instance discrimination as self-supervision improves overall performance by 2.6 points over the 46.3% CE baseline and raises accuracy across all three shot splits.The reported gains are +1.9% for many-shot, +3.3% for medium-shot, and +1.6% for few-shot classes; the feature visualization shows better separation, especially for tail classes.Source: §4.3.1 · p. 7, Figure 3 · p. 7
• III-hard (test) beats Stage-I by 2.7% and the CE baseline by 5.3%; IV-LWS adds a further 0.3%.III-hard uses the hard-label classifier; IV-LWS applies LWS to that classifier.Source: §4.3.1 · p. 7

4.3.2 Study on different self-distillation strategies

• The hybrid soft classifier outperforms single self-distillation, while coupled self-distillation does not reach the 54.1% teacher result.The authors attribute limited medium- and few-shot improvement for the coupled approach to interference between hard and soft labels; they suggest hard labels complement soft supervision.Source: §4.3.2 · p. 8, Table 5 · p. 8

4.3.3 Evaluation on self-supervised task

• On ImageNet-LT, instance-discrimination accuracy is more stable across shot splits than supervised classification, whose accuracy remains low for tail categories.The authors interpret this pattern as evidence that self-supervised learning treats images equally during training, relieving the effect of imbalanced labels.Source: §4.3.3 · p. 8, Figure 4 · p. 8

4.4. Visualization

• A larger temperature makes the distilled-label distribution flatter; the experiments use T = 2, selected by cross-validation.The authors divide logits by temperature before applying softmax and sum the outputs across training samples to calculate the distribution.Source: §4.4 · p. 8

5. Conclusion

• SSD combines distilled supervision with self-supervision-guided soft labels and reports state-of-the-art performance on ImageNet-LT, CIFAR100-LT, and iNaturalist 2018.The proposed labels leverage both top-down semantics and bottom-up data structure.Source: §5 · p. 8