PaperlayerView exact revision on arXiv

Source-linked AI summary

BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition

Boyan Zhou, Quan Cui, Xiu-Shen Wei, Zhao-Min Chen

arXiv:1912.02413v4cs.CVcs.LG

TL;DR

Long-tailed visual recognition suffers from extreme class imbalance and limited data for tail classes. BBN separates representation and classifier learning across bilateral branches with cumulative learning, achieving best results across long-tailed benchmarks, including iNaturalist.

  • Problem

    Long-tailed visual recognition remains difficult because few head classes dominate the data while most tail classes have very few samples.

  • Method

    BBN uses conventional and re-balancing branches to separately support representation and classifier learning, combined with cumulative learning that gradually emphasizes tail data.

  • Results

    BBN achieves the best results across four long-tailed benchmarks; on long-tailed CIFAR-10 with imbalance ratio 100, it reaches 20.18% error, 2.79% lower than LDAM-DRW.

  • Takeaways & Limitations

    The study shows that re-balancing promotes classifier learning but damages representation learning, motivating separate treatment of these objectives in long-tailed recognition.

Abstract

from arXiv · show

Our work focuses on tackling the challenging but natural visual recognition task of long-tailed data distribution (i.e., a few classes occupy most of the data, while most classes have rarely few samples). In the literature, class re-balancing strategies (e.g., re-weighting and re-sampling) are the prominent and effective methods proposed to alleviate the extreme imbalance for dealing with long-tailed problems. In this paper, we firstly discover that these re-balancing methods achieving satisfactory recognition accuracy owe to that they could significantly promote the classifier learning of deep networks. However, at the same time, they will unexpectedly damage the representative ability of the learned deep features to some extent. Therefore, we propose a unified Bilateral-Branch Network (BBN) to take care of both representation learning and classifier learning simultaneously, where each branch does perform its own duty separately. In particular, our BBN model is further equipped with a novel cumulative learning strategy, which is designed to first learn the universal patterns and then pay attention to the tail data gradually. Extensive experiments on four benchmark datasets, including the large-scale iNaturalist ones, justify that the proposed BBN can significantly outperform state-of-the-art methods. Furthermore, validation experiments can demonstrate both our preliminary discovery and effectiveness of tailored designs in BBN for long-tailed problems. Our method won the first place in the iNaturalist 2019 large scale species classification competition, and our code is open-source and available at https://github.com/Megvii-Nanjing/BBN.

1. Introduction

Long-tailed visual datasets contain many rare tail classes, making recognition difficult; re-balancing improves classifier learning but can impair feature representation. BBN addresses both objectives with separate branches and cumulative learning that shifts from universal patterns toward tail data.

  • Problem: Real-world visual datasets often have skewed long-tailed distributions, with head classes dominating samples and tail classes having few examples.This extreme imbalance creates substantial recognition challenges, especially for tail classes.
  • Prior methods: Class re-balancing methods primarily use resampling or cost-sensitive re-weighting to alleviate extreme training-data imbalance.These strategies adjust training by resampling examples or reweighting example losses.
  • Analysis: Validation experiments show that re-balancing significantly promotes classifier learning while damaging representation learning relative to the original data distribution.The experiments separately train representations and classifiers using plain training, re-weighting, and re-sampling.
  • Method: BBN uses separate conventional-learning and re-balancing branches to handle representation learning and classifier learning simultaneously.The conventional branch learns universal patterns, while the re-balancing branch models tail data.
  • Method: Cumulative learning automatically shifts BBN from universal features to tail data as training progresses while controlling branch updates to preserve learned universal features.The trade-off parameter α is generated by an Adaptor according to training epochs.
  • Results: Experiments on four benchmark long-tailed datasets show that BBN outperforms existing state-of-the-art methods, with validation and ablation studies supporting its design.The reported studies also support the preliminary discovery about re-balancing and the effectiveness of BBN’s tailored components.

2. Related work

Related work primarily addresses long-tailed recognition through class re-balancing, including re-sampling, re-weighting, and two-stage fine-tuning. Other explored paradigms include metric, meta-, knowledge-transfer, and mixup-based methods, which are outside or peripheral to the paper’s unified model.

  • Class re-balancing strategies: Re-sampling balances classes by over-sampling minority data or under-sampling dominant data, but duplication can overfit tails and discarding data harms generalization.Over-sampling repeats minority-class data, whereas under-sampling abandons dominant-class data.
  • Class re-balancing strategies: Re-weighting assigns larger loss weights to tail samples but struggles with large-scale real-world data and can cause optimization difficulty.Effective-number weighting was proposed instead of proportional frequency.
  • Class re-balancing strategies: Two-stage fine-tuning first trains on original imbalanced data, then applies re-balancing with a small learning rate to modify the training process.The method separates network training and re-balanced fine-tuning into two stages.
  • Other learning paradigms: Related approaches also include metric learning, meta-learning, knowledge transfer learning, and mixup-based augmentation, though these are outside or peripheral to the paper’s unified end-to-end model.Mixup convexly combines random image-label pairs, while manifold mixup performs the operation in feature space with ratios sampled from a β-distribution.

3. How class re-balancing strategies work?

The section separates deep classification into representation learning and classifier learning to explain how re-balancing methods work. Experiments on long-tailed CIFAR show that re-balancing primarily improves classifier learning, while plain cross-entropy produces stronger representations, with CE for representations and RS for classifiers achieving the lowest validation error.

  • Mechanism: The analysis decomposes a deep classifier into a feature extractor and classifier, corresponding to representation learning and classifier learning.This decomposition provides the framework for analyzing re-balancing methods’ effects on each component.
  • Experimental design: A two-stage experiment first learns feature extractors with CE, re-sampling, or re-weighting, then fixes those extractors while separately learning classifiers.The design isolates the effects of each learning manner on representations and classifiers.
  • Experimental setup: The experiments use long-tailed CIFAR-10 and CIFAR-100 datasets, including CIFAR-100-IR50 with imbalance ratio 50.The imbalance ratio is defined as the most frequent class size divided by the least frequent class size.
  • Findings: For a fixed representation-learning manner, RW and RS achieve lower classification error rates than CE because re-balancing adjusts classifier-weight updates toward test distributions.This supports the conclusion that re-balancing substantially promotes classifier learning.
  • Findings: For a fixed classifier-learning manner, CE yields better representations than the re-balancing manners, while CE representations combined with RS classifier learning achieve the lowest CIFAR-100-IR50 validation error.The study also evaluates representation generalization by transferring CIFAR-100-IR50 feature extractors to CIFAR-10-IR50 classifier experiments.

4. Methodology

BBN uses two coordinated branches to separate representation learning from classifier learning while sharing most network weights. A cumulative strategy shifts emphasis from universal representation patterns toward tail data during training, with equal branch weighting at inference.

  • Bilateral-branch structure: BBN separates representation learning and classifier learning into conventional-learning and re-balancing branches, respectively.Both branches use the same residual-network structure and differ only in their final residual block.
  • Data samplers: The conventional branch uses uniform sampling to preserve the original data distribution, whereas the re-balancing branch targets extreme imbalance and tail-class accuracy.The uniform sampler samples each training example once per epoch with equal probability.
  • Weights sharing: The branches share weights except for the last residual block, allowing conventional-branch representations to benefit re-balancing-branch learning.The shared backbone uses ResNet architectures such as ResNet-32 and ResNet-50.
  • Cumulative learning strategy: Cumulative learning weights conventional features by α and re-balancing features by 1 − α, with α decreasing as training epochs increase.This schedule first emphasizes universal patterns and gradually shifts attention toward tail data and classifier learning.
  • Inference: At inference, BBN fixes α to 0.5, sends equally weighted branch features to their classifiers, and adds the resulting logits element-wise.The network is trained end-to-end using integrated branch outputs.

5. Experiments

Experiments on long-tailed CIFAR and iNaturalist datasets show that BBN consistently outperforms competing methods, including two-stage fine-tuning and mixup strategies. Ablations further support its reversed sampling, cumulative learning schedule, preserved feature quality, and balanced classifier norms.

  • Long-tailed CIFAR: BBN achieves the best error rates across long-tailed CIFAR-10 and CIFAR-100 at imbalance ratios 10, 50, and 100.Comparisons include CE-DRW, CE-DRS, mixup, manifold mixup, and manifold mixup with two samplers.
  • Long-tailed CIFAR: 20.18% error on CIFAR-10 with imbalance ratio 100 is 2.79% lower than LDAM-DRW.This represents the reported extreme-imbalance result; the two-stage fine-tuning strategies also achieve comparable or better results than state-of-the-art methods.
  • iNaturalist: BBN outperforms competing approaches and baselines on iNaturalist 2017 and iNaturalist 2018, with and without the 2× scheduler.The datasets contain 5,089 and 8,142 categories, respectively, and the experiments use large-scale long-tailed settings.
  • Ablation studies: The reversed sampler performs best in the re-balancing branch among the evaluated sampling strategies.The comparison includes uniform sampling, which preserves the original long-tailed distribution, and balanced sampling, which gives all classes equal sampling probability.
  • Ablation studies: Parabolic decay is the best α-generation strategy, while decay schedules outperform equal weight, β-distribution, and parabolic increment.The result supports learning the conventional branch first and the re-balancing branch afterward.
  • Representation and classifier analysis: BBN-CB preserves feature quality comparable to CE, BBN-RB outperforms RW/RS, and BBN-ALL has the smallest classifier-norm standard deviation at σ = 0.148.The classifier norms for BBN-ALL are essentially equal across ten classes, whereas CE follows the long-tailed distribution and RW/RS have larger standard deviations.

6. Conclusions

The paper finds that class re-balancing improves classifier learning but somewhat harms representation learning in long-tailed recognition. It therefore proposes a Bilateral-Branch Network with cumulative learning to address both aspects.

  • Class re-balancing strategies significantly promote classifier learning but damage deep-feature representation learning to some extent.
  • The proposed Bilateral-Branch Network is designed to address representation learning and classifier learning simultaneously.
  • BBN incorporates a specific cumulative learning strategy for long-tailed recognition.

A. Additional experiments of different manners for representation and classifier learning (cf. Section 3 and Figure 2 of the paper) on large-scale datasets iNaturalist 2017 and iNaturalist 2018;

Experiments on iNaturalist 2017 and 2018 support the conjecture that re-balancing promotes classifier learning while potentially harming the universal discriminative ability of learned features. Cross-entropy yields better representations, whereas re-weighting and re-sampling produce lower classification error when representation learning is held constant.

  • Experimental setup: Experiments on iNaturalist 2017 and 2018 further test whether re-balancing improves classifier learning but damages universal feature representation.The study evaluates cross-entropy, re-weighting, and re-sampling in separate representation-learning and classifier-learning manners.
  • Classifiers: With the same representation-learning manner, re-weighting and re-sampling achieve lower classification error rates than cross-entropy.Their re-balancing operations adjust classifier-weight updates to match test distributions.
  • Representations: With the same classifier-learning manner, cross-entropy blocks consistently have lower error rates than re-weighting and re-sampling blocks.This indicates that cross-entropy obtains better features, while re-weighting and re-sampling yield inferior discriminative ability in learned deep features.
  • Consistency: These observations on the two large-scale datasets are consistent with results on long-tailed CIFAR datasets and further demonstrate the paper’s Section 3 discovery.Figure 5 reports top-1 error rates for the three learning manners on iNaturalist 2017 and 2018.

B. Affects of re-balancing strategies on the compactness of learned features

On CIFAR-10-IR50, intra-class compactness is measured by average ℓ2 distance from each class’s centroid. Re-balancing produces less compact learned features than conventional training, particularly for head classes.

  • Intra-class compactness is measured by averaging representations around each class centroid using their ℓ2 distances.Representations are normalized to ℓ2-norm 1 during training.
  • Re-balancing strategies yield larger averaged centroid distances than conventional training, especially for head classes.Larger averaged distances indicate that representations gather less closely in feature space.
  • Thus, re-balancing significantly worsens learned-feature compactness compared with conventional training.The observations further validate that re-balancing makes each class’s intra-class distribution more separable.

C. Comparisons between the BBN model and ensemble methods

BBN is compared with ensemble models across four long-tailed recognition datasets using classifiers trained with different data samplers. It achieves consistently lower error rates while requiring only a limited increase in network parameters through weight sharing.

  • Dataset comparisons: BBN is evaluated against ensemble methods on CIFAR-10-IR50, CIFAR-100-IR50, iNaturalist 2017, and iNaturalist 2018.Table 6 reports the comparison results across all four datasets.
  • Ensemble construction: The comparison ensembles combine models trained with uniform, balanced, and reversed data samplers to mimic BBN’s bilateral-branch design.The study trains three classification models with these samplers and reports ensembles pairing the uniform sampler with balanced or reversed sampling.
  • Results: BBN achieves consistently lower error rates than ensemble models on all evaluated datasets.This result is reported in Table 6 of the supplementary materials.
  • Results: BBN delivers better performance than ensemble models with only a limited increase in network parameters through weight sharing.The weight-sharing design is described in Section 4.2 of the paper.

D. Coordinate graph of different adaptor strategies for generating α

The coordinate graph shows how α changes during training across adaptor strategies, with decay strategies shifting BBN’s focus from representation learning toward classifier learning. Among them, parabolic decay performs best because it allows sufficient time to learn universal features before classifier learning.

  • D. Coordinate graph of different adaptor strategies for generating α: Figure 7 plots α against training progress for different adaptor strategies, using the same strategies as Table 4 except for the β-distribution randomness.The horizontal axis is the current epoch ratio, and the vertical axis is α.
  • D. Coordinate graph of different adaptor strategies for generating α: Decay strategies outperform non-decay strategies, while decreasing α gradually shifts BBN’s learning focus from representation learning to classifier learning.Values α > 0.5 can be viewed as emphasizing representation learning.
  • D. Coordinate graph of different adaptor strategies for generating α: Parabolic decay performs best because it gives the representation-learning layers enough epochs to learn universal features before robust classifier learning.The passage contrasts earlier-layer representation learning, which requires more epochs to converge, with later-layer classifier learning, which requires relatively fewer epochs.

E. Learning algorithm of our proposed BBN model

BBN trains each epoch by adapting the branch-mixing coefficient, sampling conventional and re-balanced examples separately, and optimizing a combined classification loss. The algorithm extracts independent branch features, forms mixed logits and softmax predictions, then updates the model parameters.

  • Epoch procedure: Each epoch first sets α with the paper’s adaptor, then draws samples using uniform and reversed samplers for the two branches.The conventional branch uses uniform sampling, while the re-balancing branch uses reversed sampling.
  • Branch processing: The network independently extracts conventional and re-balancing feature vectors, fc and fr, from their respective sampled inputs.The branches have separate parameters, θc and θr.
  • Prediction computation: BBN computes mixed logits z and predictions ˆp with Softmax(z), using the re-balancing branch feature and classifier weight in the stated algorithm.The algorithm lists z ← αWᵀr fr before applying softmax.
  • Optimization: The classification objective combines losses for conventional and re-balanced labels as L ← αE(ˆp, yc) + (1 −α)E(ˆp, yr).Model parameters are updated by minimizing L.
Loading 1912.02413v4…