Source-linked AI summary
Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification
Liuyu Xiang, Guiguang Ding, Jungong Han
TL;DR
Long-tailed distributions make CNN training difficult. LFME trains experts on less imbalanced cardinality-adjacent subsets and adaptively distills their knowledge into a unified student. The framework achieves state-of-the-art performance on long-tailed benchmarks and can further improve existing methods.
Problem
Long-tailed class distributions make CNN training difficult, creating a need for methods that address severe data imbalance.
Method
LFME trains multiple experts on cardinality-adjacent subsets and uses self-paced expert selection plus curriculum instance selection to distill their knowledge into one student.
Results
LFME achieves state-of-the-art performance on three benchmark long-tailed classification datasets and can be combined with state-of-the-art methods for further improvements.
Takeaways & Limitations
The framework provides a unified student-learning approach that transfers knowledge from multiple experts while supporting integration with existing long-tailed classification methods.
Abstract
from arXiv · showhide
In real-world scenarios, data tends to exhibit a long-tailed distribution, which increases the difficulty of training deep networks. In this paper, we propose a novel self-paced knowledge distillation framework, termed Learning From Multiple Experts (LFME). Our method is inspired by the observation that networks trained on less imbalanced subsets of the distribution often yield better performances than their jointly-trained counterparts. We refer to these models as 'Experts', and the proposed LFME framework aggregates the knowledge from multiple 'Experts' to learn a unified student model. Specifically, the proposed framework involves two levels of adaptive learning schedules: Self-paced Expert Selection and Curriculum Instance Selection, so that the knowledge is adaptively transferred to the 'Student'. We conduct extensive experiments and demonstrate that our method is able to achieve superior performances compared to state-of-the-art methods. We also show that our method can be easily plugged into state-of-the-art long-tailed classification algorithms for further improvements.
1 Introduction
Long-tailed class distributions make CNN training difficult, motivating LFME, which distills knowledge from experts trained on less imbalanced subsets into a unified student through adaptive learning schedules.
- Real-world datasets often contain few majority classes and many minority classes, making expected CNN performance difficult to achieve.
- LFME is proposed as a self-paced knowledge distillation method that can be plugged into previous long-tailed classification methods.
- Sorting classes by cardinality and splitting them into adjacent subsets produces less imbalanced distributions for training Expert Models.
- The framework trains a unified Student with self-paced expert selection and curriculum instance selection.
- Experiments on three benchmark datasets report superior performance against state-of-the-art methods, with further gains when LFME is combined with them.
2 Related Work
Prior long-tailed learning methods mainly address imbalance through resampling, reweighting, or knowledge transfer, while related few-shot and distillation methods provide neighboring approaches.
- Resampling methods oversample tail classes or undersample head classes to address data imbalance.
- Cost-sensitive loss functions reweight majority and minority instances in long-tailed learning.
- Head-to-tail transfer methods progressively learn transformations or use meta embeddings with feature memory for imbalance-related knowledge transfer.
- Few-shot learning generalizes knowledge from many-shot classes to new few-shot classes, whereas this work jointly models a continuous data-distribution spectrum.
- Knowledge distillation transfers information from teacher to student networks and has also been applied between models with identical architectures.
3 Motivation and Metrics for Evaluating Data Imbalance
The paper motivates balancing long-tailed data by measuring imbalance and testing cardinality-adjacent subsets, which are less imbalanced and can yield stronger CNN performance than joint training.
- Motivation: Training on a balanced dataset with fewer samples can sometimes outperform training on a larger long-tailed dataset.
- Metrics: Imbalance Ratio measures the ratio between the largest and smallest class sample counts.
- Metrics: The four imbalance metrics use class cardinalities and compare long-tailed distributions with uniformity, with smaller values indicating greater uniformity.
- Cardinality-adjacent subsets: Sorting classes by cardinality and grouping adjacent classes creates subsets with less severe long-tailed imbalance under the proposed measurements.
- Empirical motivation: CNNs trained on cardinality-adjacent subsets outperform jointly trained models on ImageNet-LT and Places-LT.
4 The LFME framework
LFME trains expert models on cardinality-adjacent subsets and transfers their knowledge to a unified student through two adaptive schedules: expert selection and instance selection.
- Framework overview: LFME splits classes into cardinality-adjacent subsets, trains one expert per subset, and uses expert logits and confidence to guide the student.The experts provide knowledge-transfer targets and instance-difficulty cues.
- Self-paced Expert Selection: As student performance approaches the experts, the self-paced scheme decreases their guidance so the experts do not become performance ceilings.The final distillation loss is an automatically weighted sum across experts.
- Self-paced Expert Selection: The student receives weighted distillation losses from all experts, with weights determined by the performance gap between each expert and the student.Top-1 validation accuracy is used to measure the gap, and weights are updated after each epoch.
- Curriculum Instance Selection: Curriculum Instance Selection uses each corresponding expert’s confidence as an instance-difficulty signal and replaces cross-entropy with a soft selection-weighted loss.A higher instance weight indicates soft selection, while a smaller weight indicates soft unselection.
- Curriculum Instance Selection: The curriculum increases selection weights monotonically from an initial subset-specific value toward 1, adding harder samples over training.Selection is organized within cardinality-adjacent subsets to begin with balanced coverage and reach all samples by the final epoch.
- Training: LFME combines the two adaptive schedules into a final loss and is trained after experts using class-level random sampling.The two schedule weights control instance selection and expert distillation.
5 Experiments
Experiments on ImageNet-LT, Places-LT, and CIFAR100-LT evaluate LFME against long-tailed baselines and isolate its adaptive learning components. LFME is competitive or superior overall, combines effectively with other methods, and benefits from self-paced scheduling.
- Experimental Settings: Experiments cover ImageNet-LT, Places-LT, and CIFAR100-LT, using many-shot, medium-shot, and few-shot subsets for the first two datasets.ImageNet-LT and Places-LT use three cardinality-adjacent subsets with thresholds {20, 100}; CIFAR100-LT uses controllable exponential-decay imbalance.
- Main Results: LFME achieves superior or comparable results to state-of-the-art methods on ImageNet-LT and Places-LT, with the largest benefits generally occurring for many-shot categories.Few-shot classes also improve and perform similarly to re-weighting methods; LFME+OLTR outperforms previous methods by a large margin.
- Main Results: LFME achieves comparable performance with LDAM on CIFAR100-LT, while combining LFME and LDAM further improves both many-shot and few-shot subsets.The reported experiment uses CIFAR100-LT with imbalance ratio 100.
- Ablation Study: Knowledge distillation improves instance-level and class-level sampling baselines by 11.3% and 3.2%, respectively, while slightly reducing few-shot accuracy.Class-level sampling improves few-shot performance but slightly decreases many-shot performance relative to instance-random sampling.
- Ablation Study: Self-paced Expert Selection yields 0.4% and 1.3% overall gains in the evaluated settings, and Curriculum Instance Selection adds 1.0% few-shot accuracy.The curriculum component compensates for the few-shot decrease associated with knowledge distillation.
- Ablation Study: Self-paced Expert Selection consistently distills many-shot and medium-shot experts, reduces cross-entropy loss, and decays few-shot expert weight after the student exceeds that expert.The visualization also reports that the linear growing schedule performs best, while α values are generally robust except α = 1.0, which reduces performance.
6 Conclusions
LFME addresses long-tailed classification by training multiple experts on less long-tailed subsets and adaptively distilling their knowledge into a unified student. Experiments and visualizations support its effectiveness and state-of-the-art performance on long-tailed benchmarks.
- LFME trains several expert models on cardinality-adjacent, less long-tailed subsets before distilling their knowledge into a unified student model.
- Two levels of adaptive learning transfer expert knowledge to the student through self-paced learning.
- Extensive experiments and visualizations verify LFME’s effectiveness and that of each component on long-tailed classification benchmarks.