PaperlayerView exact revision on arXiv

Source-linked AI summary

Distilling Virtual Examples for Long-tailed Recognition

Yin-Yin He, Jianxin Wu, Xiu-Shen Wei

arXiv:2103.15042v3cs.CV

TL;DR

Long-tailed recognition leaves tail categories under-represented and less accurate. DiVE distills teacher predictions as virtual examples, and reports the best results on long-tailed benchmarks, including large-scale iNaturalist.

  • Problem

    Long-tailed training data leave tail categories with few examples and lower accuracy, and prior approaches may not let categories interact.

    Source: §1 · p. 1

  • Method

    DiVE trains a teacher and distills its predictions, treated as virtual examples, into a student while tuning the virtual-example distribution toward flatness.

    Source: §1 · pp. 1–2, §3 · p. 3

  • Results

    DiVE achieved the best results on long-tailed benchmarks, including large-scale iNaturalist; on iNaturalist2018, top-1 accuracy was at least 1 percentage point higher than all compared methods in both settings.

    Source: §4.3 · p. 7, §5 · p. 8

  • Takeaways & Limitations

    Virtual examples allow head-category predictions to contribute to tail-category recognition, and the paper reports that distilling them can alleviate extreme imbalance, particularly for tail classes.

    Source: §1 · p. 1, §5 · p. 8

    At very large temperatures, the supervision signal becomes roughly uniform and contains little information for distillation.

    Source: §3.4 · p. 5

Abstract

from arXiv · show

We tackle the long-tailed visual recognition problem from the knowledge distillation perspective by proposing a Distill the Virtual Examples (DiVE) method. Specifically, by treating the predictions of a teacher model as virtual examples, we prove that distilling from these virtual examples is equivalent to label distribution learning under certain constraints. We show that when the virtual example distribution becomes flatter than the original input distribution, the under-represented tail classes will receive significant improvements, which is crucial in long-tailed recognition. The proposed DiVE method can explicitly tune the virtual example distribution to become flat. Extensive experiments on three benchmark datasets, including the large-scale iNaturalist ones, justify that the proposed DiVE method can significantly outperform state-of-the-art methods. Furthermore, additional analyses and experiments verify the virtual example interpretation, and demonstrate the effectiveness of tailored designs in DiVE for long-tailed problems.

1. Introduction

Long-tailed data leave tail categories with few samples and lower accuracy, while existing approaches often lack cross-category interaction. DiVE treats teacher predictions as virtual examples and distills from a flatter distribution.

1. Introduction

  • Long-tailed datasets concentrate training images in head categories, leaving tail categories with few samples and lower accuracy.The introduction contrasts naturally long-tailed data with roughly balanced, manually built datasets.Source: §1 · p. 1
  • 61.58% test accuracy on CIFAR-100-LT with imbalance factor 100 was more than 10 percentage points above existing long-tail methods.The student was distilled on CIFAR-100-LT from a teacher trained on the complete CIFAR-100 training set.Source: §1 · p. 1
  • Teacher predictions act as virtual examples, so a prediction can contribute to a class different from the input’s ground-truth category.For a dog prediction of (0.7, 0.3), the paper interprets the output as 0.7 dog and 0.3 cat virtual examples; such contributions let head categories help recognize tail categories.Source: §1 · pp. 1–2
  • 39.20%, 43.25%, and 53.71% were the student accuracies using CE, BSCE, and FULL virtual-example distributions, respectively; flatter teacher distributions aligned with higher accuracy.Figure 1 compares the original input distribution with virtual-example distributions across Many, Medium, and Few categories.Source: §1 · p. 2, Figure 1 · p. 2
  • The paper argues that removing tail-category bias requires a virtual-example distribution flatter than the original input distribution.It reports that making the distribution more balanced increases student accuracy, while τ = 3 gives FULL a 61.58% accuracy in the cited experiment.Source: §1 · p. 2
  • DiVE proposes to level the virtual-example distribution and distill from it, and the authors report large-margin gains over existing methods across long-tailed datasets.The introduction presents this as a method for explicitly learning from a balanced virtual-example distribution.Source: §1 · p. 2

2. Related Work

Prior long-tail methods rebalance or transfer knowledge in ways that can leave categories interacting independently or require complex designs. DiVE instead uses knowledge distillation over virtual examples to support cross-category interaction.

2. Related Work

  • Resampling can cause overfitting or limit generalization, and resampling or reweighting methods sacrifice head accuracy to compensate for tail classes.The cited resampling variants include minority oversampling and majority undersampling.Source: §2 · p. 2
  • Decoupling representation and classifier learning improves long-tailed performance, but its gains are confined to the classifier because tail features remain under-represented..Source: §2 · p. 2
  • Some head-to-tail transfer methods use meta-learning, memory banks, or expert ensembles; other approaches require extra data or longer training and can be hard to generalize.DiVE is presented as a simpler knowledge-distillation pipeline.Source: §2 · p. 2
  • Resampling, reweighting, and decoupled training operate independently within categories, while DiVE’s virtual-example distillation enables categories to interact.The paper frames this interaction as examples from different categories naturally helping one another.Source: §2 · pp. 2–3
  • DiVE differs from prior long-tail uses of knowledge distillation by aiming to flatten the teacher’s virtual-example distribution.The paper relates its interpretation of distillation to deep label distribution learning, while distinguishing it from treating distillation as learnable label smoothing.Source: §2 · p. 3

3. Distilling the Virtual Examples

DiVE treats teacher predictions as virtual examples and distills them into a student, with the aim of making their class distribution flatter than the input distribution. The section connects this interpretation to label distribution learning and describes how temperature and power normalization adjust flatness.

3. Distilling the Virtual Examples

  • DiVE trains a teacher on the long-tailed task, then distills its virtual examples into a student; its loss combines a BSCE term with a temperature-scaled KD term.The teacher signal can be leveled with temperature and power normalization, while the student signal uses temperature without power normalization.Source: §3 · p. 3, §3.4 · p. 6

3.1. KD ≈DLDL

  • At temperature τ = 1, knowledge distillation is equivalent to deep label distribution learning (DLDL) with the effective ground-truth label distribution ˜t.Because the effective target has zero gradients with respect to student parameters, its KL-divergence loss is equivalent for student training.Source: §3.1 · p. 3, §3.2 · p. 4

3.2. From label distributions to virtual examples

  • A teacher prediction assigns virtual-example mass across classes, so a dog image's non-negligible cat score can contribute a virtual cat example despite the input being a dog.The distillation loss equals the summed cross-entropy losses over these virtual examples.Source: §3.2 · p. 4

3.3. The virtual example distribution must be flat

  • As the virtual-example distribution flattens, tail accuracy increases significantly while head accuracy remains almost intact; a balanced long-tailed model requires more virtual examples for tail classes than their input counts.The two-class CIFAR-10 experiment used one class with 5,000 training samples and another with 500.Source: §3.3 · p. 5
  • Existing category-weighting methods such as BSCE and LWS only indirectly flatten virtual-example distributions, motivating explicit adjustment.The section also identifies the beneficial degree of flatness as an open design question.Source: §3.3 · p. 5

3.4. Level and distill virtual examples

  • Increasing teacher temperature makes its virtual-example distribution more balanced, but very large temperatures yield nearly uniform signals with little information.The rule of thumb selects a relatively small temperature for which tail average virtual-example counts are slightly above head counts; the stated CIFAR-100-LT example chooses τ = 6.Source: §3.4 · pp. 5–6, Figure 5 · p. 5

4. Experimental Results

DiVE is evaluated on CIFAR-100-LT, ImageNet-LT, and iNaturalist2018, with analyses of its components and a RIDE-based variant. The reported results show strong comparisons across datasets and accuracy subsets.

4.1. Datasets and implementations

  • Experiments cover CIFAR-100-LT, ImageNet-LT, and iNaturalist2018, using ResNet-32, ResNeXt-50, and ResNet-50 backbones, respectively.The CIFAR-100-LT experiments use imbalance factors β = 10, 50, and 100.Source: §4.1 · p. 6

4.2. Evaluation setups and comparison methods

  • Evaluation reports overall top-1 accuracy and Many-, Medium-, and Few-shot accuracy; iNaturalist2018 also includes top-5 accuracy.The compared methods include cross-entropy and focal-loss baselines, balanced softmax, De-confund-TDE, RIDE, and RIDE-DiVE.Source: §4.2 · pp. 6–7

4.3. Main results

  • DiVE is best across CIFAR-100-LT imbalance factors 10–100, beating De-confund-TDE and LFME by 1.55 percentage points over LFME.Table 1 reports top-1 accuracy.Source: §4.3 · p. 7, Table 1 · p. 7
  • On ImageNet-LT, DiVE obtains almost consistently higher accuracy than compared methods across Many, Medium, Few, and All categories.Table 2 reports the ImageNet-LT comparison.Source: §4.3 · p. 7, Table 2 · p. 7
  • On iNaturalist2018, DiVE’s top-1 accuracy is at least 1 percentage point higher than every compared method in both training settings; 200 epochs add 2.58% top-1 and 1.54% top-5 accuracy.Table 3 reports results for 90 and 200 epochs.Source: §4.3 · p. 7, Table 3 · p. 7
  • RIDE-DiVE’s student surpasses both its teacher and RIDE, despite the BSCE teachers being much weaker than RIDE’s teacher on two datasets.The method uses a six-expert RIDE teacher to generate virtual examples for a four-expert student.Source: §4.3 · p. 7, Table 5 · p. 8

4.4. Further Analyses

  • Using only the DLDL loss reduces DiVE accuracy by less than 1% in all three cases, yet remains better than most compared methods.The comparison concerns removing the BSCE loss term.Source: §4.4 · p. 8
  • Power normalization and higher temperature make virtual examples more even and improve accuracy; DiVE’s tτ variant outperforms variants using ˜t, while its distribution is balanced with the tail slightly above the heads.The ground-truth-included variant without BSCE has the worst results and most imbalanced distribution; row #4 has the lowest accuracy among tτ variants.Source: §4.4 · p. 8, Figure 6 · p. 8

5. Conclusions

DiVE distills teacher predictions as virtual examples, using head-to-tail category interactions to address long-tailed recognition. The paper reports strong benchmark results and identifies long-tailed detection as future work.

5. Conclusions

  • DiVE distills teacher predictions as virtual examples into a student, leveraging interactions between categories, including unrelated head and tail categories.The paper says this can alleviate extreme imbalance, particularly for tail classes.Source: §5 · p. 8
  • The authors report that DiVE achieved the best results on long-tailed benchmarks, including large-scale iNaturalist.They also describe a rule of thumb for adjusting the virtual-example distribution toward flatness to improve distillation accuracy.Source: §5 · p. 8
  • The authors identify extending DiVE to long-tailed detection as future work.Source: §5 · p. 8

A. Additional experiments on the influence of virtual example distribution flatness

Additional experiments vary category selection, imbalance factor, and dataset size to examine virtual-example distribution flatness. The reported pattern is rising tail accuracy with head accuracy nearly intact as the distribution becomes flatter.

A. Additional experiments on the influence of virtual example distribution flatness

  • The additional experiments vary the categories and imbalance factor to test the conjecture that virtual-example distributions should be flat.The results are presented in Figure 7.Source: A. Additional experiments on the… · p. 11
  • As the virtual-example distribution gets flatter, tail accuracy increases significantly while head accuracy remains almost intact across different imbalance factors or dataset sizes.The experiments also include dissimilar airplane and dog categories.Source: A. Additional experiments on the… · p. 11
  • The reported observations are consistent with the conjecture that the virtual-example distribution should be flat.Source: A. Additional experiments on the… · p. 11

B. Implementation details

Implementation settings vary across the three datasets, including backbones, training schedules, and DiVE parameters. RIDE-DiVE uses a separate expert-based distillation setup.

B. Implementation details

  • CIFAR-100-LT uses ResNet-32, 200 epochs, τ = 3, power normalization p = 0.5, and α = 0.5.Training uses batch size 128; the learning rate starts at 0.1, warms up for five epochs, and decays at epochs 120 and 160.Source: B. Implementation details · p. 11
  • ImageNet-LT uses ResNeXt-50 for 90 epochs with τ = 9 and α = 0.5, without power normalization.Training follows the strategies of [17].Source: B. Implementation details · pp. 11–12
  • iNaturalist2018 uses ResNet-50 with τ = 2, power normalization p = 0.5, and α = 0.5; results are reported for 90 and 200 epochs.For RIDE-DiVE, a six-expert RIDE teacher is distilled into each expert of a four-expert student.Source: B. Implementation details · p. 12

C. Results on various shifted test label distributions

The evaluation tests models on shifted ImageNet-LT label distributions and reports that DiVE surpasses CE and BSCE by a large margin, whether target-distribution information is known or not.

C. Results on various shifted test label distributions

D. t-SNE visualization

On CIFAR100-LT (β = 100), t-SNE visualizations indicate that DiVE produces more compact, more separable embeddings than CE, while RIDE-DiVE performs better than both.

D. t-SNE visualization

  • DiVE enlarges inter-class variance and reduces intra-class variance for both head and tail classes, making similar-appearance classes easier to separate.The passage gives mouse and squirrel as an example.Source: D. t-SNE visualization · p. 12
  • CE embeddings are dispersed for both head and tail classes, complicating distinction between classes with similar appearances.The passage exemplifies this with mouse and squirrel.Source: D. t-SNE visualization · p. 12
  • RIDE-DiVE performs better than both CE and DiVE in the embedding-space comparison.Source: D. t-SNE visualization · p. 12

E. Sample images visualization

The ImageNet-LT examples compare CE, BSCE, and DiVE predictions on selected test images where DiVE is correct, illustrating corrections to both semantically nearby and distant categories.

E. Sample images visualization

  • The examples were selected from ImageNet-LT test images on which DiVE predicts correctly, to illustrate its prediction corrections.Source: E. Sample images visualization · p. 12
  • DiVE corrects predictions to both semantically nearby and distant categories in the selected ImageNet-LT examples.Examples include Polyporus frondosus and Siberian husky for nearby categories, and pot and ski mask for distant categories.Source: E. Sample images visualization · p. 12
  • The figure presents each image with CE, BSCE, and DiVE predictions; category colors mark Many, Medium, and Few classes, and DiVE predictions are ground-truth labels.Source: Figure 9 · p. 13
Loading 2103.15042v3…