PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning a Deep ConvNet for Multi-label Classification with Partial Labels

Thibaut Durand, Nazanin Mehrasa, Greg Mori

arXiv:1902.09720v1cs.CV

TL;DR

The paper addresses scalable multi-label classification when only partial annotations are available, since exhaustive clean labels are costly to collect. It compares labeling strategies, proposes a proportion-aware loss and curriculum-based missing-label prediction, and reports improved performance with an accurate Bayesian-uncertainty curriculum strategy across large-scale multi-label settings.

  • Problem

    Clean, exhaustive multi-label annotations are difficult to scale, motivating learning when only some labels are known for each image.

  • Method

    The paper compares labeling strategies, introduces a loss that exploits each image’s known-label proportion, and uses curriculum learning with GNN-based category correlations to predict missing labels.

  • Results

    The experiments show that the proposed loss significantly improves performance and that Bayesian uncertainty is an accurate strategy for labeling missing labels.

  • Takeaways & Limitations

    Partial labels can support scalable end-to-end multi-label ConvNet training, while curriculum-based missing-label prediction can improve the available training annotations.

Abstract

from arXiv · show

Deep ConvNets have shown great performance for single-label image classification (e.g. ImageNet), but it is necessary to move beyond the single-label classification task because pictures of everyday life are inherently multi-label. Multi-label classification is a more difficult task than single-label classification because both the input images and output label spaces are more complex. Furthermore, collecting clean multi-label annotations is more difficult to scale-up than single-label annotations. To reduce the annotation cost, we propose to train a model with partial labels i.e. only some labels are known per image. We first empirically compare different labeling strategies to show the potential for using partial labels on multi-label datasets. Then to learn with partial labels, we introduce a new classification loss that exploits the proportion of known labels per example. Our approach allows the use of the same training settings as when learning with all the annotations. We further explore several curriculum learning based strategies to predict missing labels. Experiments are performed on three large-scale multi-label datasets: MS COCO, NUS-WIDE and Open Images.

1. Introduction

The paper studies scalable multi-label classification when only some labels are known per image. It compares partial-label annotation strategies, introduces a proportion-aware loss for ConvNets, and predicts missing labels with curriculum learning and category correlations.

  • Motivation: Multi-label datasets represent scenes with several objects, but collecting consistent and exhaustive labels for every image is difficult to scale.Partial labels offer a scalable alternative because only some categories need to be annotated for each image.
  • Partial labels: Partial annotations record known labels while leaving other category labels unknown, such as knowing an image contains a car and not a bear but not knowing about a person, boat, or apple.This setting differs from treating every unknown category as absent.
  • Labeling strategy: With a fixed label budget, partially annotating all images performs better than fully annotating a small subset.The paper empirically compares alternative labeling strategies for multi-label datasets.
  • Learning with partial labels: The proposed scalable ConvNet method introduces a loss that generalizes binary cross-entropy by exploiting the proportion of known labels per image.The loss adapts automatically to each image’s known-label proportion and retains the training settings used with complete labels.
  • Missing-label prediction: The paper predicts missing labels using curriculum learning, progressively adding selected predictions, and improves predictions by modeling correlations between categories with a GNN.The approach addresses the fact that multi-label categories are not independent.

2. Related Work

Prior work addresses missing multi-label annotations through negative-label assumptions, label propagation, matrix completion, or curriculum learning. The paper positions its approach as addressing scalability and ConvNet fine-tuning limitations in these methods.

  • Missing labels: Many missing-label methods treat unknown labels as negative, effectively converting multi-label learning with missing labels into fully labeled learning.Webly supervised approaches commonly use the assumption that only the queried category is present and other categories are absent.
  • Label completion: Other methods propagate information through label-label and instance-instance correlations using matrix completion or low-rank regularization.These approaches complete the instance-label matrix from observed relationships.
  • Scalability: Most related methods cannot fine-tune a deep ConvNet because they require the training set in memory rather than mini-batch optimization.This limits transfer from pre-trained architectures, for which fine-tuning is important.
  • Problem setting: Partial-label learning differs from semi-supervised learning because every image has some labels, whereas semi-supervised learning leaves some examples entirely unlabeled.The paper also distinguishes its setting from ambiguously labeled learning.
  • Curriculum learning: Curriculum learning starts with easy samples or subtasks and gradually increases difficulty, while prior self-paced methods select easy samples as they update model parameters.Related work also includes learned curricula for noisy data and curriculum approaches designed for multi-class rather than multi-label classification.
  • Never-Ending Learning: The approach is related to Never-Ending Learning, which uses previously learned knowledge to improve models and collect or discover structured datasets.Prior examples apply this paradigm to object-class models, language, and visual knowledge.

3. Learning with Partial Labels

The paper trains ConvNets with partial labels using a proportion-aware loss, category-correlation modeling, and curriculum-based missing-label prediction. Its design targets scalable, end-to-end learning while adapting to varying amounts of annotation.

  • Overview: The method combines a partial-label loss, a GNN for category correlations, and curriculum learning to predict missing labels.The ConvNet is first trained on clean partial labels, then missing labels are added progressively using model predictions.
  • Partial-BCE loss: The partial-BCE loss normalizes by the proportion of known labels and ignores categories marked unknown.Unlike standard BCE, it gives each example the same importance regardless of how many labels are known.
  • Partial-BCE loss: The normalization function is designed to match standard BCE when all labels are present, requiring g(1) = 1.Its hyperparameters control how normalization changes with the known-label proportion, including inverse-proportion weighting as a special case.
  • GNN classification: The GNN represents each category as a node in a fully connected graph and initializes node states from ConvNet outputs.Shared message and hidden-state update functions propagate information among categories; the hidden-state update uses a GRU.
  • Missing-label prediction: The curriculum objective alternates optimization of ConvNet parameters and selected-label variables, beginning with clean partial labels.Missing labels are added only when classification scores exceed a threshold, and selected labels are represented by binary variables.
  • Missing-label prediction: Missing-label strategies include score thresholds, fixed score proportions, positive-only prediction, ensemble scores, and Bayesian uncertainty.The ensemble strategy averages model scores, while the uncertainty strategy uses score variance to estimate pair difficulty.

4. Experiments

Experiments evaluate annotation strategies, partial-BCE, GNN-based category modeling, missing-label curricula, and partial-BCE design choices across multi-label datasets. Results favor distributing clean labels across images, using proportion-aware loss normalization, and combining curriculum relabeling with uncertainty and GNN reasoning.

  • 4.1. What is the best strategy to annotate a dataset?: With a fixed clean-label budget, partially annotating all images outperforms fully annotating a smaller image subset, especially at lower label proportions.The comparison uses the same number of clean labels across strategies; performance increases logarithmically with the proportion of labels.
  • 4.1. What is the best strategy to annotate a dataset?: A noisy+ strategy retaining one positive label per image introduces 2.4% noisy labels and reduces performance by about 7pt versus the clean baseline.When an image has multiple positive labels, one is randomly retained and the others are switched to negative labels.
  • 4.2. Learning with partial labels: Partial-BCE significantly improves MAP over standard BCE across datasets, with larger gains at lower known-label proportions and a 4 pt advantage on Open Images.The loss exploits label-proportion information while retaining the same training setting as learning with all annotations.
  • 4.2. Learning with partial labels: GNNs improve performance at each label proportion on MS COCO and also improve Open Images results when the label proportion is small.The experiments support modeling category correlations, while noting that ConvNets also learn some correlations implicitly through shared representations.
  • 4.3. What is the best strategy to predict missing labels?: The bayesian uncertainty curriculum is the best missing-label strategy; predicting all missing labels at once decreases MAP because it introduces too much label noise.The threshold strategy beats the proportion strategy, ensembles do not significantly improve over a single model, and predicting only positive labels performs poorly.
  • 4.4. Method analysis: For partial-BCE normalization, g(0.1) around 20 performs best, values from 3 to 50 are similar, and γ = 1 outperforms γ = −1.The normalization is important and robust; using a large normalization value drops performance, while γ = 3 is better for large label proportions but slightly worse for small proportions than γ = −1.
  • 4.4. Method analysis: Relabeling significantly raises the 0-1 exact match score by +5pt and Micro-F1 by +2.5pt in the 10%-known-label MS COCO ablation.Fine-tuning, partial-BCE, GNN, and relabeling each contribute, and the contributions are complementary.

5. Conclusion

The paper presents a scalable end-to-end approach for learning multi-label classifiers from partial labels. Its loss improves performance, while Bayesian-uncertainty curriculum learning accurately predicts missing labels.

  • The approach enables scalable end-to-end learning of multi-label classifiers with partial labels.
  • The proposed loss function significantly improves performance.
  • Bayesian-uncertainty curriculum learning accurately predicts missing labels.

A.1. Multi-label classification with GNN

The GNN receives ConvNet category outputs, propagates information across a fully connected category graph, and predicts using both initial and final hidden states after three iterations.

  • The GNN takes the ConvNet output x as input, with C representing the number of categories.
  • The fully connected message function averages information from all graph nodes except the current node.
  • Each graph node is initialized with its corresponding ConvNet output, then messages and hidden states are updated iteratively.
  • The final prediction uses both the first and last hidden states, with T = 3 iterations in the experiments.

A.2. Experimental details

Experiments use several large-scale multi-label datasets with different category counts and annotation conditions, including a strongly partially labeled Open Images setting.

  • Experiments use Pascal VOC 2007, MS COCO, NUS-WIDE, and Open Images.
  • NUS-WIDE contains 269,648 Flickr images annotated with 81 visual concepts, with 150k randomly sampled training images.
  • Open Images experiments use human labels for 600 boxable classes, with only 0.9% of training labels available.
  • Models are trained with SGD for 20 epochs using 448 × 448 images and random horizontal flips as augmentation.

A.3. Multi-label metrics

The paper evaluates multi-label predictions with overall, macro-, micro-, exact-match, and per-class metrics. These metrics differ in whether they weight samples or categories equally.

  • Zero-one exact-match accuracy counts a prediction as correct only when all labels are correctly predicted.
  • Per-class measures treat classes equally regardless of sample size, so rare-class performance can dominate the score.
  • Overall precision and recall treat all samples equally regardless of class.
  • Macro-F1 averages F1 scores across categories and is more sensitive to rare-category performance.
  • Micro-F1 computes performance over predictions considered as a whole and is more affected by major categories.

A.4. Analysis of the initial set of labels

The model is robust to which labels are initially selected in the partial-label setting, across label proportions and evaluation metrics on MS COCO val2014.

  • Across four random seeds, performance is robust to the initial set of partial labels for every tested label proportion and metric.Experiments use MS COCO val2014 with a ResNet-101 WELDON.
  • The evaluation covers MAP, M-F1, m-F1, 0-1 exact match, and PC-P across varying label proportions.

A.5. Analysis of the labeling strategies

Partially annotating all images generally outperforms completely annotating a smaller subset under a fixed label budget, while noisy labels substantially reduce performance.

  • Labeling strategies: For a given label proportion, partial labels outperform complete image labels across network architectures, with larger gains at lower proportions.
  • Metrics: Partial labels outperform complete image labels for MAP, Macro-F1, and Micro-F1, whereas complete image labels perform better on 0-1 exact match.Exact match requires every category prediction to be correct, favoring complete image labels.
  • Noisy supervision: Noisy+ labeling reduces performance on every metric across Pascal VOC 2007, MS COCO, and NUS-WIDE relative to training with 10% clean labels.

A.6. Comparison of the loss functions

The partial-BCE loss improves several metrics over BCE, remains advantageous across learning rates and low label proportions, and is combined with missing-label prediction in the final model.

  • Loss comparison: Partial-BCE significantly improves MAP, 0-1 exact match, Macro-F1, and Micro-F1 over BCE on MS COCO and Pascal VOC 2007.The improvement is larger when the known-label proportion is lower.
  • Loss comparison: Partial-BCE and BCE are equivalent at 100% known labels, while partial-BCE remains better across a broad learning-rate range and small label proportions.
  • Hyperparameters: A learning rate of 0.02 increases performance and produces a monotone increase with label proportion, although the optimal learning rate depends on the dataset.
  • Hyperparameters: Normalization values g(0.1) between 3 and 50 significantly improve performance across the tested network architectures on MS COCO.
  • Final model: The final model combines partial-BCE, a GNN, and missing-label prediction, outperforming two BCE baselines on most metrics.
  • Final model: The final model has significantly better 0-1 exact match than the complete-label BCE baseline, but lower overall precision and substantially higher overall recall.
Loading 1902.09720v1…