PaperlayerView exact revision on arXiv

Source-linked AI summary

Asymmetric Loss For Multi-Label Classification

Emanuel Ben-Baruch, Tal Ridnik, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, Lihi Zelnik-Manor

arXiv:2009.14119v4cs.CVcs.LG

TL;DR

Multi-label classification is dominated by many negative labels relative to positives, making it difficult to preserve useful positive gradients. The paper introduces ASL, which treats positive and negative samples differently through asymmetric focusing and probability shifting, and reports stronger results across major benchmarks with a simple, efficient setup. Its dynamic asymmetry scheme offers easier hyper-parameter control but can slightly reduce performance early in training.

  • Problem

    Multi-label classification has severe positive-negative imbalance, while effective prior approaches may require complex architectures or external information.

  • Method

    ASL decouples positive and negative focusing and shifts negative probabilities to discard easy negatives and possibly mislabeled samples.

  • Results

    ASL outperforms common losses and prior state-of-the-art methods across MS-COCO, Pascal-VOC, NUS-WIDE, and Open Images.

  • Takeaways & Limitations

    ASL is effective and easy to use with standard architectures, without increasing training or inference time or requiring external information.

  • Takeaways & Limitations

    The dynamically tuned scheme achieves mAP scores 0.2% below the best fixed-γ− ASL score, possibly because early training is highly influential.

Abstract

from arXiv · show

In a typical multi-label setting, a picture contains on average few positive labels, and many negative ones. This positive-negative imbalance dominates the optimization process, and can lead to under-emphasizing gradients from positive labels during training, resulting in poor accuracy. In this paper, we introduce a novel asymmetric loss ("ASL"), which operates differently on positive and negative samples. The loss enables to dynamically down-weights and hard-thresholds easy negative samples, while also discarding possibly mislabeled samples. We demonstrate how ASL can balance the probabilities of different samples, and how this balancing is translated to better mAP scores. With ASL, we reach state-of-the-art results on multiple popular multi-label datasets: MS-COCO, Pascal-VOC, NUS-WIDE and Open Images. We also demonstrate ASL applicability for other tasks, such as single-label classification and object detection. ASL is effective, easy to implement, and does not increase the training time or complexity. Implementation is available at: https://github.com/Alibaba-MIIL/ASL.

1. Introduction

Multi-label classification faces severe positive-negative imbalance and incomplete ground-truth labels, while prior solutions often require complex architectures or external information. The paper proposes ASL, a simple loss-based approach that separates positive and negative treatment and improves benchmark performance.

  • Existing approaches exploit label correlations, knowledge priors, image parts, and attentional regions to improve multi-label classification.
  • Recent methods often require extensive architecture modifications or external information, motivating a simpler loss-centered solution.The proposed alternative uses standard architectures and training schemes.
  • Multi-label images contain few positive labels among many possible negatives, creating inherent positive-negative imbalance.Missing ground-truth labels are also common in multi-label datasets.
  • ASL decouples positive and negative modulation, down-weights easy negatives, and hard-thresholds negatives that may be mislabeled.Its probability shifting can also discard very hard negative samples suspected as mislabeled.
  • ASL compares favorably with cross-entropy and focal loss, producing significant mAP improvements while balancing positive and negative sample probabilities.The method also includes dynamic adjustment of asymmetry during training through a fixed gap between average positive and negative probabilities.
  • Across four popular benchmarks, ASL reaches 86.6% mAP on MS-COCO, surpassing the previous top result by 2.8%.The solution uses standard architectures, adds no training or inference time, and requires no external information.

2. Asymmetric Loss

ASL addresses multi-label imbalance by separately controlling positive and negative contributions, combining asymmetric focusing with probability shifting. Gradient and probability analyses motivate its ability to suppress easy negatives while preserving positive-sample influence.

  • Asymmetric Focusing: Multi-label training faces a trade-off: stronger focal focusing can suppress easy-negative gradients but also eliminate gradients from rare positive samples.
  • Asymmetric Focusing: ASL decouples positive and negative focusing parameters, typically using γ− > γ+ to control their loss contributions separately.
  • Asymmetric Probability Shifting: Probability shifting hard-thresholds very easy negatives by setting pm = max(p − m, 0), fully discarding them when p < m.
  • Asymmetric Loss: ASL combines asymmetric focusing and probability shifting, providing soft attenuation of easy negatives and hard thresholding through the probability margin m.
  • Gradient Analysis: Gradient analysis attributes ASL’s combined asymmetries to hard thresholding, nonlinear attenuation, rejection of possibly mislabeled negatives, and continuous loss gradients.
  • Probability Analysis: Probability analysis shows symmetric losses create negative probability gaps, whereas ASL eliminates the gap and can dynamically adjust asymmetry toward a target probability difference.

3. Experimental Study

Experiments on MS-COCO compare focal loss, asymmetric focusing, asymmetric probability margins, combined asymmetries, and adaptive asymmetry. ASL’s combined components perform best, while adaptive asymmetry improves over focal loss but remains slightly below the best fixed scheme.

  • Focal Loss Vs Cross-Entropy: 84.0% vs 85.1% mAP: focal loss outperforms cross-entropy on MS-COCO, with its best scores at 2 ≤ γ ≤ 4.Lower γ provides insufficient down-weighting of easy negatives, whereas higher γ overly down-weights rare positive samples.
  • Asymmetric Focusing: Decreasing γ+ improves mAP for fixed γ− values of 2 and 4, with γ+ = 0 producing the best experimental results.The authors relate this to maintaining larger gradient magnitudes for positive samples.
  • Asymmetric Probability Margin: Asymmetric probability margins improve mAP for both cross-entropy and focal loss, with cross-entropy performing best at the low margin m = 0.05.For focal loss, the optimal margin is higher at 0.3 ≤ m ≤ 0.4.
  • Comparing Different Asymmetries: Combining asymmetric focusing and probability margins produces the best results, enabling easy-sample discarding, nonlinear attenuation, and rejection of possibly mislabeled hard negatives.Static weighting is insufficient for the high negative-positive imbalance, whereas ASL dynamically handles easy and hard samples.
  • Adaptive Asymmetry: 85.8% vs 85.1% mAP: adaptive asymmetry improves over focal loss when demanding the unbiased case Δp_target = 0.A higher target gap, Δp_target = 0.2, performs even better and gives extra focus to rare positive samples.
  • Adaptive Asymmetry: The dynamic scheme’s top mAP scores remain 0.2% below the best fixed-γ− ASL score, so the experiments use fixed asymmetry thereafter.The authors attribute the small degradation partly to early-training effects that make dynamic tuning initially sub-optimal.

4. Dataset Results

ASL is evaluated across four multi-label datasets and additional vision tasks, consistently outperforming common loss functions and prior methods. The experiments also show that ASL preserves efficiency while benefiting from modern backbones, pretraining, and higher input resolution.

  • Experimental scope: ASL evaluation covers four multi-label datasets and tests applicability to single-label classification and object detection.The datasets are MS-COCO, Pascal-VOC, NUS-WIDE, and Open Images.
  • MS-COCO: On MS-COCO, ASL significantly outperforms prior state-of-the-art methods on ResNet101 and improves the top mAP score by more than 1%.Other reported metrics also improve.
  • Efficiency and architecture: ASL requires no architecture modifications and does not increase training or inference time, while remaining complementary to more complex methods.Combining ASL with those methods could improve scores but would increase training complexity and reduce throughput.
  • MS-COCO: ASL outperforms focal loss and cross-entropy across OFA-595, ResNet101, and TResNet-L backbones.This demonstrates robustness to backbone selection and superiority over the compared loss functions.
  • MS-COCO: Better ImageNet pretraining increases MS-COCO mAP by almost 2%, while increasing input resolution from 448 to 640 further improves results.The comparison considers ImageNet-1K versus ImageNet-21K pretraining.
  • Pascal-VOC: ASL achieves new state-of-the-art results on Pascal-VOC with and without additional pretraining, and outperforms cross-entropy and focal loss.The Pascal-VOC comparison uses the same training settings as MS-COCO.
  • NUS-WIDE and Open Images: ASL reaches new state-of-the-art results on NUS-WIDE and significantly outperforms focal loss and cross-entropy on Open Images.The Open Images result supports applicability to large-scale, extreme-classification settings.
  • Additional tasks: ASL outperforms focal loss on relevant single-label classification and object detection datasets.These tasks were selected because they contain substantial background or long-tail cases and are known to benefit from focal loss.

5. Conclusion

The paper presents ASL as an asymmetric loss that treats positive and negative samples differently. Analysis and experiments show balanced sample probabilities and improved performance across major multi-label benchmarks.

  • Conclusion: ASL uses two complementary asymmetric mechanisms that operate differently on positive and negative samples.The paper examines the loss derivatives and proposes adaptive adjustment of asymmetry during training.
  • Conclusion: ASL outperforms common loss functions and prior state-of-the-art methods on MS-COCO, Pascal-VOC, NUS-WIDE, and Open Images.The conclusion attributes this result to extensive experimental analysis.

Appendices

The appendix reports adaptive asymmetry dynamics during training for a target probability shift of 0.1.

  • Adaptive Asymmetry Dynamics: Figure 9 tracks γ− and ∆p throughout training with ∆p_target = 0.1, while γ+ = 0 and m = 0.05.The figure presents the dynamics of the adaptive asymmetry settings under these fixed conditions.

B. Multi-Label General Training Details

The general training procedure uses a fixed 60-epoch Adam schedule with standard augmentation and simple RGB normalization.

  • Training procedure: Models are trained for 60 epochs with Adam, a 1-cycle policy, and a maximum learning rate of 2e-4.Standard augmentation techniques are used for regularization.
  • Normalization: RGB channels are scaled to [0,1] instead of using common ImageNet-statistics normalization.The authors report that ImageNet normalization does not improve results in their experiments.

C. Comparing MS-COCO On All Common Metrics

The paper compares ASL with established methods on MS-COCO and Pascal-VOC, and with alternative losses on NUS-WIDE. These sections organize benchmark comparisons across common metrics and loss functions.

  • C. Comparing MS-COCO On All Common Metrics: ASL is compared with known state-of-the-art models on MS-COCO across all common metrics.Results use input resolution 448.
  • C. Comparing MS-COCO On All Common Metrics: ASL is compared with other loss functions on Pascal-VOC using percentage-based metrics.

F. Open Images Training Details

The Open Images experiment uses a reduced downloaded test set and treats untagged labels as negative with reduced weights. Training proceeds at two input resolutions with separate training and fine-tuning stages.

  • F. Open Images Training Details: 114,648 Open Images test images were downloaded because of missing Flickr links, covering about 5,400 unique tagged classes.
  • F. Open Images Training Details: Untagged Open Images labels are treated as negative with reduced weights to address partial labeling.
  • F. Open Images Training Details: The network is trained for 30 epochs at input resolution 224 and fine-tuned for 5 epochs at resolution 448.

G. Fine-Grain Single-Label Classification Results

The paper evaluates ASL beyond multi-label classification on fine-grained single-label recognition and reports comparisons using the Herbarium dataset. It also identifies the dataset’s long-tailed scale and competition metric.

  • G. Fine-Grain Single-Label Classification Results: Herbarium 2020 contains over 1M images representing over 32,000 plant species in a long-tailed fine-grained classification setting.
  • G. Fine-Grain Single-Label Classification Results: The paper also reports comparison tables for MS-COCO and Pascal-VOC, plus model and loss comparisons for NUS-WIDE.
  • G. Fine-Grain Single-Label Classification Results: The Herbarium competition uses macro F1 as its official metric.
  • G. Fine-Grain Single-Label Classification Results: ASL outperforms focal loss by a large margin on the Herbarium dataset.The comparison is reported on an unseen private test set.
  • G. Fine-Grain Single-Label Classification Results: The ASL test-set score would have placed third among 153 teams in the Herbarium 2020 competition.

H. Object Detection Results

For object detection, the paper replaces only FCOS’s classification focal loss with ASL while retaining the bounding-box and centerness losses. ASL improves detection mAP over standard focal loss.

  • H. Object Detection Results: The object-detection experiment uses MS-COCO’s 118k-image training set and 5k-image evaluation set with FCOS and ATSS enhancements.
  • H. Object Detection Results: FCOS uses classification, bounding-box, and centerness losses; only its classification focal loss is replaced with ASL.The bounding-box component uses IoU loss, while centerness uses plain cross-entropy.
  • H. Object Detection Results: 0.4% improvement in mAP is obtained with ASL over standard focal loss on MS-COCO object detection.
Loading 2009.14119v4…