PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Label Learning from Single Positive Labels

Elijah Cole, Oisin Mac Aodha, Titouan Lorieul, Pietro Perona, Dan Morris, Nebojsa Jojic

arXiv:2106.09708v2cs.CVcs.LG

TL;DR

Exhaustive multi-label annotation is costly and often leaves false negatives, motivating learning from only one positive label per image. The paper extends losses and introduces missing-label estimation and positive-count regularization for this setting. Across experiments, it finds that classification can be competitive with full supervision using far fewer labels. The approach is nevertheless bounded by assumptions about how positives are selected and by memory costs for its label estimator.

  • Problem

    Exhaustive multi-label annotation is difficult and can leave false negatives because relevant labels are missed or unobserved.

  • Method

    The paper extends existing losses and introduces missing-label estimation and expected-positive regularization for single-positive training.

  • Results

    The paper reports classification results competitive with full label supervision using an order of magnitude fewer labels.

  • Takeaways & Limitations

    The findings open avenues for efficient crowdsourcing of annotations for large-scale multi-label datasets.

  • Takeaways & Limitations

    The annotation simulation assumes each present positive is equally likely to be selected, although human annotators may favor familiar or salient objects.

Abstract

from arXiv · show

Predicting all applicable labels for a given image is known as multi-label classification. Compared to the standard multi-class case (where each image has only one label), it is considerably more challenging to annotate training data for multi-label classification. When the number of potential labels is large, human annotators find it difficult to mention all applicable labels for each training image. Furthermore, in some settings detection is intrinsically difficult e.g. finding small object instances in high resolution images. As a result, multi-label training data is often plagued by false negatives. We consider the hardest version of this problem, where annotators provide only one relevant label for each image. As a result, training sets will have only one positive label per image and no confirmed negatives. We explore this special case of learning from missing labels across four different multi-label image classification datasets for both linear classifiers and end-to-end fine-tuned deep networks. We extend existing multi-label losses to this setting and propose novel variants that constrain the number of expected positive labels during training. Surprisingly, we show that in some cases it is possible to approach the performance of fully labeled classifiers despite training with significantly fewer confirmed labels.

1. Introduction

Multi-label annotation is costly and prone to false negatives because images can contain multiple relevant classes that annotators may miss. This paper studies the extreme case of many images with only one positive label each and presents methods that substantially reduce supervision while retaining classification performance.

  • Motivation: Multi-label annotation is expensive because annotators must identify all applicable classes, while rare or difficult-to-detect objects are often missed.These omissions create false negatives in training data.
  • Result: 20 times fewer labels still yielded comparable performance to full supervision for LROLE on PASCAL VOC 2012.The figure compares test MAP for one-positive training with fully labeled LBCE training.
  • Problem: The paper studies single positive multi-label learning, where each training image has one observed positive label and no confirmed positives or negatives.This setting favors many minimally annotated images over exhaustive annotation of fewer images.
  • Motivation: This setting can reduce annotation costs, address multi-class images containing multiple classes, and test performance at the minimal limit of supervision.The paper notes that standard missing-label approaches often require more observed labels than this setting provides.
  • Contributions: The paper extends existing multi-label methods, proposes a loss that estimates missing labels during training, and evaluates multiple losses across four datasets.The evaluation covers the single positive setting and compares alternative supervision regimes.

2. Related Work

The paper situates single-positive learning within partially observed and positive-unlabeled multi-label classification. Unlike many prior approaches, its setting provides only one positive label per image and no confirmed negatives.

  • Existing settings: Multi-label classification predicts multiple concepts per image, but exhaustive supervision is difficult to acquire in practice.Prior work addresses missing labels through negative assumptions, ignoring labels, reconstruction, correlations, generative models, and label cleaning.
  • Missing-label assumptions: The paper focuses on unobserved labels that may be positive or negative but are incorrectly treated as absent under strong assumptions.This differs from conventional label noise because the issue arises from incomplete observation.
  • Single-positive setting: Single-positive annotations arise when images contain multiple relevant objects but only one class is recorded, including settings beyond vision.The paper gives multi-class image classification and species distribution modeling as examples.
  • Relation to prior work: The setting is closest to positive-unlabeled learning but is distinguished by having one positive label per item and no explicit in-depth prior treatment of this case.When multiple positives are available, class co-occurrence information can be inferred, which is not directly possible here.

3. Problem Statement

The paper formalizes multi-label prediction with partially observed label vectors and isolates the single-positive case, where one positive label is known and all others are unknown. It asks how to design losses that narrow the risk gap relative to fully observed training.

  • Formal setting: In multi-label classification, each input has a binary vector indicating which of L classes are relevant.The classifier predicts a probability for each class rather than a single normalized class distribution.
  • Formal setting: The classifier is trained by minimizing a surrogate multi-label loss over observed examples.The surrogate is chosen because it is easier to optimize than the target risk directly.
  • Partial observation: Partially observed labels use z_n in {0,1,∅}, where ∅ means the corresponding true label may be either absent or present.The training data therefore do not expose the complete label vector y_n.
  • Single-positive case: The single-positive case observes exactly one positive label per training example, with all remaining labels unknown.Its formal entries are restricted to 1 or ∅, with one indicator equal to 1 for each example.
  • Research question: The central objective is to design a partially observed loss that minimizes the risk difference between partial and fully observed training.The paper frames fully observed training as having no greater risk than partial training.

4. Multi-Label Learning

The paper contrasts losses for fully observed, partially observed, and positive-only labels, emphasizing that positive-only training requires assumptions about unobserved labels. Ignoring them degenerates, while assuming they are negative introduces false negatives.

  • Losses by supervision regime: Binary cross-entropy is presented as the baseline loss for fully observed multi-label data, with variants adapted to incomplete supervision.These variants trade weaker observation requirements for stronger assumptions about P(y_i|x_n).
  • Notation: The classifier outputs one probability f_ni for each class, and these probabilities need not sum to one across classes.This reflects the independent binary structure of the BCE formulation.
  • Partially observed labels: The ignore-unobserved loss sets loss terms for unobserved labels to zero.Its interpretation is that unobserved labels are already perfectly predicted.
  • Partially observed labels: Ignore-unobserved losses are unsuitable for positive-only data because, without negative examples, they collapse to an always-predict-positive solution.The paper nevertheless uses an ignore-unobserved-negatives variant as an experimental intermediate with access to true negatives.
  • Positive-only labels: The assume-negative loss treats every unobserved label as negative, making it applicable to positive-only data but introducing false negatives.This is described as a common approach to the positive-only setting.

5. Learning From Only Positive Labels

The paper develops ways to learn multi-label classifiers when only one positive label is observed and unobserved labels may be false negatives. It combines loss modifications, expected-positive regularization, and online label estimation to reduce false-negative damage while retaining useful supervision.

  • Motivation: False negatives can significantly reduce performance even though unobserved labels are usually negative, motivating four mitigation strategies.The proposed strategies are weak negatives, label smoothing, expected positive regularization, and online label estimation.
  • Loss modifications: WAN reduces the influence of assumed negative labels through a weight γ between zero and one.γ = 1 recovers standard BCE, while γ = 0 permits the trivial always-positive solution; the paper uses γ = 1/(L −1).
  • Loss modifications: Label smoothing replaces binary target distributions with softened targets to reduce the impact of incorrectly assumed negative labels.The paper applies smoothing independently to each of the L binary target distributions and uses ϵ = 0.1.
  • Expected positive regularization: Expected positive regularization constrains batch predictions toward an expected number k of positive labels per image, avoiding the always-positive solution.The regularizer uses normalized deviation by L and is combined with a hyperparameter λ; batch-level regularization allows images to contain different numbers of positives.
  • Online label estimation: ROLE jointly trains an image classifier and an online label estimator, combining estimated-label BCE with expected-positive regularization.The method is described as similar to expectation-maximization, with the classifier matching estimated labels while respecting known positives and the expected positive count.
  • Online label estimation: The proposed training framework is motivated by the observation that convolutional networks fit informative labels faster than uninformative random labels.This provides intuition for jointly estimating missing labels and training the classifier.

6. Experiments

Experiments evaluate single-positive multi-label learning across four datasets using fixed-feature linear classifiers and end-to-end fine-tuning. LROLE is strongest in the linear setting, while label smoothing, initialization, and loss choice materially affect performance.

  • Experimental setup: Experiments cover four benchmark datasets using fixed-feature linear classifiers and end-to-end fine-tuned networks.The backbone is an ImageNet-pretrained ResNet-50.
  • Experimental setup: Training data are created by retaining one randomly selected positive label per image from fully labeled datasets.The same retained label set is used for all comparisons within each dataset.
  • Results: LROLE is the strongest linear-case method, often approaching or surpassing LIUN despite using substantially fewer labels.In the fine-tuned case, better initialization substantially benefits both LROLE and LAN.
  • Results: LROLE reaches VOC12 test MAP comparable to fully labeled LBCE while using 20 times fewer labels in the linear case.This comparison concerns the VOC12 linear-classifier experiment.
  • Results: LAN falls farther behind fully supervised LBCE after end-to-end fine-tuning, whereas LAN−LS and LROLE perform better in most cases.The text attributes the larger fine-tuning gap to false negatives corrupting the backbone feature extractor.
  • Results: LROLE recovers more of the full unobserved label matrix than LAN−LS, while LAN−LS remains a strong simple baseline.LROLE requires more training-time parameters but adds no inference-time computational overhead.
  • Results: Linear initialization substantially improves end-to-end training for LROLE and LAN−LS.The recommended procedure freezes the backbone initially before fine-tuning the entire network.

7. Limitations

The simulated annotation process assumes every present positive label is equally likely to be selected, while LROLE requires storing an N × L label matrix. These assumptions limit realism and scalability.

  • Annotation simulation assumes each present positive label is equally likely to be selected.The paper leaves annotator preferences, such as familiarity or object saliency, for future work.
  • LROLE requires online estimation and memory storage of an N × L label matrix.For ImageNet, storing the full matrix would require 4GB of memory, and larger datasets or label sets could make this infeasible.

8. Conclusion

The paper studies single positive training as an underexplored form of partially observed multi-label classification. It reports competitive classification with full supervision using an order of magnitude fewer labels and identifies several future directions.

  • Single positive training can achieve classification results competitive with full label supervision using an order of magnitude fewer labels.
  • The findings motivate future work on efficient crowdsourcing for large-scale multi-label datasets.
  • Future directions include semi-supervised multi-label classification, self-supervised representation learning, and more general partially observed multi-label settings.

A. Results for Additional Losses

Additional-loss experiments show that straightforward pairwise ranking does not beat LAN, while asymmetric label smoothing changes performance differently for fine-tuned and linear models.

  • Pairwise ranking: The straightforward pairwise-ranking adaptation does not outperform the LAN baseline in the single-positive setting.
  • Asymmetric label smoothing: Asymmetric label smoothing independently modulates positive and negative labels through ϵp and ϵn.LAN−LS(0,0.1) smooths negative labels while leaving positive labels untouched.
  • Asymmetric label smoothing: +1.0 MAP for VOC12 and +0.7 MAP for COCO occur with LAN−LS(0,0.1) versus LAN−LS in the fine-tuned case.
  • Asymmetric label smoothing: -0.3 MAP for VOC12 and -0.4 MAP for COCO occur with LAN−LS(0,0.1) versus LAN−LS in the linear case.

B. Effect of Misspecified k

The appendix evaluates estimating k from small fully labeled samples rather than using its true value. Performance is reported as stable across the reasonable interval estimated from only five examples.

  • LROLE + LinearInit performance is stable for k values in the interval estimated from five fully labeled examples.
  • Performance is likely to be similar when using k* or an estimate k̂5 based on only five fully labeled training examples.
  • The study estimates k from M ∈ {5, 10, 25} fully labeled examples using 10^5 trials and approximate 95% confidence intervals.

C. Dataset Statistics

Table A3 summarizes dataset label statistics, including the maximum, minimum, and average number of positive labels per image. The true k is the fully labeled training-set average used for LROLE and LEPR in the main paper.

  • The average number of positives per image is the true k used for LROLE and LEPR in the main paper.The effect of choosing other k values is studied separately in Section B.
  • Table A3 reports each dataset’s maximum, minimum, and average number of positive labels per image.The average is computed on the fully labeled training set.
  • Figure A1 evaluates LROLE across 10 linearly spaced k values from the interval I5 in Table A2.For VOC12, results summarize five runs; for COCO, each value uses one run.
Loading 2106.09708v2…