Source-linked AI summary
Calibrated Disambiguation for Partial Multi-label Learning
Zhuoming Li, Yuheng Jia, Mi Yu, Zicong Miao
TL;DR
Partial multi-label learning must disambiguate over-annotated candidate labels, while existing curriculum methods lack adaptive weighting guidance. PML-CD derives an optimal weighting function equivalent to confidence calibration, learns a transferable calibrator, and adds prototype alignment; it outperforms baselines in 15 of 16 reported conditions.
Problem
PML datasets contain candidate labels that mix relevant and irrelevant labels, while existing curriculum-based methods lack adaptive weighting and theoretical guidance for optimal weights.
Source: Introduction · p. 1, Curriculum Learning · p. 3Method
PML-CD derives an optimal curriculum weight equivalent to confidence calibration, predicts weights from positive-label confidence histograms with a transferable calibrator, and adds prototype alignment regularization.
Source: Introduction · p. 2Results
PML-CD outperforms baseline models in 15/16 training conditions, although it is slightly inferior under low-noise conditions on VOC.
Source: Comparison Results · p. 6Takeaways & Limitations
The optimal weighting formulation provides guidance for curriculum design, while the calibrator is reported to provide high-quality calibration on unseen datasets and different training conditions.
Source: Introduction · p. 2, Conclusion · p. 7Heuristic adaptive weighting remains challenging because optimal curricula lack a mathematical formulation and some predictors depend on assumptions that may not apply.
Source: Curriculum Learning · p. 3
Abstract
from arXiv · showhide
Partial multi-label learning (PML) aims to train a classifier on datasets whose instances are over-annotated with not only relevant labels but also irrelevant labels, which is common when datasets are collected from crowd-sourcing platform. Existing works primarily approach it from a curriculum learning perspective, leveraging the memorization effect to disambiguate noisy labels and produce robust predictions. However, these methods are based on non-adaptive weighting functions and lack theoretical guidance for optimal weighting. To overcome these issues, a calibrated disambiguation model named PML-CD is proposed. We first formulate the optimal weighting function for curriculum-based disambiguation, which is equivalent to the calibration of the model's predicted confidences, thus providing guidance for curriculum designing. To obtain the optimal weighting function from PML dataset during the training, a transferable calibrator is designed, which takes the histogram of positive samples' confidences as input, and outputs the optimal curriculum weighting for training. Prototype alignment regularization is also proposed to promote the model's performance. Experiments conducted on Pascal VOC, MS-COCO, NUS-WIDE and CUB have verified that our method outperforms existing state-of-the-art PML methods.
Introduction
PML addresses over-annotated examples containing both relevant and irrelevant candidate labels. PML-CD formulates curriculum weighting as confidence calibration, predicts adaptive weights from confidence histograms, and adds prototype alignment regularization.
Introduction
- PML instances carry candidate labels that include both relevant and irrelevant labels, often because crowd-sourced annotation is imprecise.The resulting task is to disambiguate corrupted labels while learning a robust classifier.Source: Introduction · p. 1
- Existing curriculum weighting functions are heuristic, and the suitable weighting varies across datasets, noise rates, and training epochs.The paper notes that existing curriculum-based PML methods do not adapt weights to training feedback.Source: Introduction · p. 1
- PML-CD formulates the optimal sample weight as TP/(TP + FP), equivalent to calibrating predicted confidence.The formulation provides an upper bound for curriculum-based disambiguation.Source: Introduction · pp. 1–2
- Because TP/FP status is unknown during PML training, a transferable calibrator predicts optimal weights from positive-label confidence histograms.The calibrator is data-driven and is described as producing fine-grained predictions on unseen datasets.Source: Introduction · p. 2
- Prototype alignment regularization is proposed to support better multi-label representations and improve robustness.The paper describes the regularizer as promoting separable and discriminative features.Source: Introduction · p. 2
Related Work
Related work covers multi-label image architectures, PML disambiguation assumptions, and curriculum learning for noisy labels. The paper identifies challenges in designing adaptive weighting strategies and notes risks from early relabeling.
Multi-label Image Classification
- Multi-label image classification methods use CNNs with recurrent, graph-based, propagation, and Transformer architectures to exploit label correlations.The cited examples include recurrent networks, graph convolutional networks, graph propagation, and Transformers (Liu et al. 2021a).Source: Multi-label Image Classification · p. 2
Partial Multi-label Learning
- PML methods use manifold and label-correlation assumptions, low-rank constraints, or combinations of these approaches to separate ground-truth labels from noise.Recent deep-network image PML methods mainly use memory effects and small-loss criteria for curriculum-guided disambiguation.Source: Partial Multi-label Learning · p. 2
Curriculum Learning
- Curriculum learning uses training difficulty to weight examples progressively, with weighting functions and adaptation strategies including confidence histograms, statistics, and loss rankings.Prior work includes hard, linear, logarithmic, and Gaussian weighting forms, with different parameter-updating schemes.Source: Curriculum Learning · p. 3
- Heuristic adaptive weighting is challenging because optimal curricula lack mathematical formulation, some feature constructions are suboptimal, and generative predictors rely on potentially inapplicable distribution assumptions.Curriculum-based relabeling also risks incorrectly correcting difficult but accurate labels early in training, which could damage further learning.Source: Curriculum Learning · p. 3
Preliminaries
Preliminaries define PML training data as candidate labels that may include noise and describe curriculum-based disambiguation through confidence-dependent weighting. The optimal weight is the posterior probability that an observed label is correct, linking curriculum weighting to confidence calibration.
Preliminaries
- PML training uses corrupted candidate labels ỹ_i that include relevant labels and additional noise, while y_i denotes the ground-truth labels.Source: Preliminaries · p. 3
- Curriculum learning uses predicted confidence and the memorization effect to weight training losses, favoring samples expected to be clean over corrupted samples.Source: Preliminaries · p. 3
- The optimal curriculum weight is P(y_ij = ỹ_ij | t_ij), which models the probability that an observed label is correct given its predicted confidence.The paper identifies this posterior weighting as a theoretical upper bound for small-loss-guided disambiguation.Source: Preliminaries · p. 4
- The paper frames optimal weighting as confidence calibration and proposes evaluating weighting quality with calibration metrics such as ECE.Source: Figure 2 · p. 3, Preliminaries · p. 4
Methods
PML-CD uses warm-up training followed by confidence-calibrated disambiguation with prototype alignment. Its calibrator maps confidence histograms to label reliabilities, while the two training losses encourage separable and discriminative representations.
Methods
- During warm-up, PML-CD trains on the raw dataset with BCE; disambiguation then uses calibrated label reliabilities as weighted-BCE weights alongside prototype alignment.Both phases run until early stopping.Source: Methods · p. 4
- A CNN backbone and Transformer with class-specific query vectors produce per-class regional representations, followed by corresponding linear projections for confidence predictions.This follows the ML-Decoder architecture.Source: Methods · p. 4
Calibrator
- The calibrator estimates label reliability from binned confidences of partial labels, separating true-positive and false-positive groups and interpolating a confidence–reliability relationship.The method uses 20 confidence bins and can form that relationship individually for each class.Source: Calibrator · p. 4
- A data-driven calibrator maps confidence-bin frequencies to reliability estimates, avoiding assumptions that TP and FP confidences follow particular distributions.It is trained from confidence statistics collected across epochs and training procedures with different noise rates, then can be loaded as a plugin.Source: Calibrator · pp. 4–5
Prototype Alignment Regularization
- Classification loss encourages positive and negative examples to be linearly separable, while prototype alignment clusters them around class prototypes for more discriminative representations.Prototype alignment is applied during disambiguation, when the representations are initialized well enough for meaningful prototypes.Source: Prototype Alignment Regularization · p. 5
Experiments
PML-CD is evaluated on four real-world image datasets under injected label noise, with comparisons spanning multi-label, traditional PML, and curriculum-based methods. It outperforms baselines in 15 of 16 conditions, while ablations examine its calibrator and prototype alignment regularization.
Settings
- Experiments use MS-COCO 2014, Pascal VOC 2007, NUS-WIDE, and CUB 200, with CUB’s 312 binary attributes treated as labels and noise injected by flipping zero labels.The noise rate ρ varies across experimental conditions.Source: Settings · p. 6
- Baselines include multi-label classifiers, traditional PML methods, and curriculum-based PML methods.Traditional PML methods use ImageNet-pretrained ResNet-101 embeddings to support image classification.Source: Settings · p. 6
- Models use an ImageNet-pretrained ResNet-101 backbone, 224 × 224 images, and early stopping; performance is evaluated with mAP.The learning rate varies by dataset: 1 × 10−4 for VOC and CUB, and 1 × 10−5 for COCO and NUS-WIDE.Source: Settings · p. 6
- The calibrator is pre-trained on VOC data spanning noise rates from 10% to 90%, using per-class confidence and reliability histograms collected across training epochs.It is trained by minimizing MSE.Source: Settings · p. 6
Comparison Results
- PML-CD outperforms baseline models in 15 of 16 conditions, with stronger performance under high noise but slightly worse performance on low-noise VOC.It also performs well on unseen COCO, NUS-WIDE, and CUB datasets.Source: Comparison Results · pp. 6–7, Table 1 · p. 7
Ablation Study
- The weighting-policy ablation evaluates ECE and mAP on MS-COCO at 80% noise, comparing the pre-trained and optimal calibrators with hard, soft, and identity weighting.The supplied results passage identifies the comparison but does not report the metric values.Source: Ablation Study · p. 7, Table 2 · p. 7
- Prototype alignment regularization improves performance, especially under high-noise conditions.The ablation compares no regularization, regularization during disambiguation, and regularization during both warm-up and disambiguation.Source: Ablation Study · p. 7, Table 3 · p. 7
Conclusion
PML-CD applies curriculum-based disambiguation through calibration. It derives an optimal weighting function, learns an adaptive calibrator from confidence histograms, and adds prototype alignment regularization; comparison experiments verify its effectiveness.
Conclusion
- PML-CD applies curriculum-based disambiguation through calibration, deriving an optimal weighting function and using a calibrator for adaptive weighting across training conditions.Prototype alignment regularization is also proposed to improve robustness during training.Source: Conclusion · p. 7