PaperlayerView exact revision on arXiv

Source-linked AI summary

Can multi-label classification networks know what they don't know?

Haoran Wang, Weitang Liu, Alex Bocchieri, Yixuan Li

arXiv:2109.14162v2cs.LG

TL;DR

OOD detection for multi-label classification is underexplored, even though models may encounter inputs unrelated to their training labels. The paper proposes JointEnergy, which aggregates label-wise energies and gives a joint-likelihood interpretation. Across three benchmarks, it consistently outperforms maximum-score-based methods and reduces FPR95 by up to 10.05%, establishing state-of-the-art performance.

  • Problem

    OOD uncertainty estimation for multi-label classification remains underexplored, requiring joint use of information across labels rather than reliance on a dominant label.

  • Method

    JointEnergy aggregates label-wise energy scores across all labels to estimate joint OOD uncertainty without directly fitting a generative joint-likelihood model.

  • Results

    JointEnergy outperforms existing methods across MS-COCO, PASCAL-VOC, and NUS-WIDE, reducing FPR95 by up to 10.05% against the best baselines.

  • Takeaways & Limitations

    Aggregating energies across labels provides better separation between in-distribution and OOD inputs than using information from only one label.

  • Takeaways & Limitations

    The joint-likelihood interpretation assumes conditionally independent labels, although experiments report superior performance without imposing that condition.

Abstract

from arXiv · show

Estimating out-of-distribution (OOD) uncertainty is a central challenge for safely deploying machine learning models in the open-world environment. Improved methods for OOD detection in multi-class classification have emerged, while OOD detection methods for multi-label classification remain underexplored and use rudimentary techniques. We propose JointEnergy, a simple and effective method, which estimates the OOD indicator scores by aggregating energy scores from multiple labels. We show that JointEnergy can be mathematically interpreted from a joint likelihood perspective. Our results show consistent improvement over previous methods that are based on the maximum-valued scores, which fail to capture joint information from multiple labels. We demonstrate the effectiveness of our method on three common multi-label classification benchmarks, including MS-COCO, PASCAL-VOC, and NUS-WIDE. We show that JointEnergy can reduce the FPR95 by up to 10.05% compared to the previous best baseline, establishing state-of-the-art performance.

1 Introduction

OOD detection for multi-label classification remains underexplored despite its relevance to real-world applications, because uncertainty must combine information across labels rather than rely on one dominant output. The paper proposes JointEnergy, which aggregates label-wise energies and achieves state-of-the-art results across three multi-label benchmarks.

  • Motivation: Multi-label OOD detection addresses test-time inputs with no label intersection with training classes, a setting common in applications where images have multiple labels.The paper identifies medical imaging as one motivating example.
  • Method: JointEnergy estimates OOD uncertainty by jointly aggregating label-wise energy scores instead of using only the largest model output.This captures information from multiple dominant labels and is interpreted from a joint likelihood perspective.
  • Results: 10.05% lower FPR95 was achieved on MS-COCO with a DenseNet against ImageNet OOD data than the best-performing baselines.Consistent improvements were also observed on PASCAL-VOC, NUS-WIDE, and an alternative architecture.
  • Results: Up to 51.93% degradation in FPR95 occurred on MS-COCO when logits were simply summed across labels instead of using summed energies.The ablation indicates that the label-wise scoring function and aggregation method must be chosen compatibly.
  • Resources: The study contributes three multi-label evaluation tasks from MS-COCO, PASCAL-VOC, and NUS-WIDE and releases code and data for reproducible research.These resources support future evaluation of multi-label OOD detection.

2 Background

Multi-label classification allows each instance to have multiple labels, while OOD detection seeks to distinguish in-distribution inputs from inputs belonging to an unrelated distribution. The background introduces energy-based scoring as a foundation for OOD uncertainty estimation.

  • Multi-label Classification: Multi-label classification assigns an instance a subset of labels represented by a binary vector, with a shared-feature neural network producing the outputs.End-to-end shared-feature training is described as more computationally efficient than training completely independent classifiers.
  • Out-of-distribution Detection: OOD detection defines a decision function that identifies whether a test input comes from the in-distribution marginal D_in or an out-of-distribution distribution D_out.An OOD input is described as having no label in common with the in-distribution data.
  • Energy Function: In energy-based OOD detection, classifier logits define energies and a Boltzmann transformation defines the corresponding probability distribution.The multi-class formulation interprets the energy of class y_i as E(x, y_i) = −f_yi(x).

3 Method

JointEnergy estimates multi-label OOD uncertainty by aggregating label-wise energy scores and interprets the result through joint likelihood. It treats higher aggregate energy as indicating in-distribution data and lower energy as indicating OOD data.

  • JointEnergy: Label-wise free energy captures uncertainty for one label but does not capture uncertainty jointly across labels.JointEnergy addresses this limitation by combining information across labels.
  • JointEnergy: JointEnergy is a scoring function that aggregates label-wise energy scores across all labels to estimate joint OOD uncertainty.The method is designed for inputs that can have several labels, unlike single-label energy estimates.
  • JointEnergy: JointEnergy uses the sum of label-wise energies, with negation producing a score where larger values indicate in-distribution data.The label-wise energies are negative by definition, so the aggregation convention reverses their sign.
  • Mathematical Interpretation: Under a conditional-independence assumption, JointEnergy can be interpreted using joint conditional likelihood and data density.The assumption facilitates the theoretical interpretation but is not imposed in the experiments.
  • Mathematical Interpretation: The joint-likelihood interpretation increases discrimination because OOD data is expected to have lower joint conditional likelihood across the labels.The method avoids directly estimating the joint likelihood with generative models, which can be computationally intractable to train and optimize on multi-label datasets.
  • OOD Detection: The OOD indicator compares JointEnergy with a threshold τ, classifying higher-energy inputs as in-distribution and lower-energy inputs as OOD.The threshold can be selected so that a high fraction, such as 95%, of in-distribution data is correctly classified.

4 Experiments

Experiments evaluate JointEnergy across three multi-label datasets and show consistent OOD-detection gains over maximum-score and alternative aggregation methods. Ablations and qualitative examples indicate that summing label-wise energies captures useful multi-label information while preserving practical usability.

  • Experimental setup: Experiments use MS-COCO, PASCAL-VOC, and NUS-WIDE as in-distribution datasets, with ImageNet and Textures evaluated as OOD data.Models use DenseNet-121 classifiers trained separately for the three datasets, and performance is measured with FPR95, AUROC, and AUPR.
  • Overall results: JointEnergy outperforms the best-performing baseline across all three multi-label classifiers and reduces MS-COCO FPR95 by 10.05% compared with MaxLogit.AUROC curves report threshold-dependent behavior, and consistent improvement is also observed with a ResNet architecture.
  • Practical considerations: JointEnergy is hyperparameter-free and avoids training a separate OOD classifier required by approaches such as Mahalanobis distance.The paper notes that Mahalanobis assumes class-conditional Gaussian feature representations, which may be poorly suited to multi-label joint distributions.
  • Aggregation ablations: MaxEnergy does not outperform JointEnergy because using only the largest label-wise energy discards information from other labels.MaxEnergy is approximately close to MaxLogit when label logits are large, while JointEnergy is interpretable from a joint-likelihood perspective.
  • Aggregation ablations: Summing prior label-wise scores can worsen OOD detection: SumLogit increases MS-COCO FPR95 from 43.53% with MaxLogit to 95.46%.The degradation occurs because logits mix positive and negative values, whereas JointEnergy uses a compatible energy function for summation.
  • Comparisons: JointEnergy reduces MS-COCO FPR95 by 11.56% versus SumProb and improves AUROC over SumProb by 3.38%, 4.57%, and 4.48% on MS-COCO, PASCAL-VOC, and NUS-WIDE.These comparisons show gains against a prior aggregation-based score across all three datasets.

5 Related Work

Related work spans multi-label classification, deep-learning OOD uncertainty estimation, generative OOD detection, and energy-based learning. The paper situates JointEnergy within efforts to detect unknown inputs without relying solely on generative density estimation.

  • Multi-label Classification: Multi-label classification identifies multiple classes or tags within an input, a setting motivated by images containing several objects and associated labels.Prior work applies convolutional networks to image tagging across datasets including NUS-WIDE and road-scene imagery.
  • OOD Uncertainty Estimation: Deep-learning OOD research addresses neural-network overconfidence on OOD data using methods including ODIN and Mahalanobis distance.These methods form the prior OOD-detection context in which JointEnergy is introduced.
  • Generative-based OOD Detection: Generative OOD detectors estimate in-distribution density, but deep generative models can assign high likelihood to OOD data.This motivates continued interest in alternative OOD-detection approaches.
  • Energy-based Learning: Energy-based learning provides a framework spanning probabilistic and non-probabilistic models, including discriminative classifiers and OOD uncertainty estimation.Prior work established energy scores as a tool for OOD detection.

6 Conclusion and Outlook

The paper proposes JointEnergy for multi-label OOD uncertainty estimation and reports improved separation and state-of-the-art detection performance.

  • JointEnergy aggregates energy scores across all labels for multi-label OOD uncertainty estimation.
  • Aggregating energies produces better separation between in-distribution and OOD inputs than using information from only one label.
  • JointEnergy has a joint-likelihood interpretation and outperforms competitive baseline methods.

7 Societal Impact

The project aims to improve the dependability and trustworthiness of multi-label classification models used in real-world applications.

  • The work targets more dependable and trustworthy multi-label classification models.
  • Potential application areas include digital content understanding, driver assistance and autonomous vehicles, and unseen disease identification.
  • The authors release code and hope to provide researchers with a methodological perspective and practitioners with an easy-to-use tool for OOD data.

A Evaluation on different architecture

Additional evaluation uses ResNet-based classifiers and compares energy-based OOD approaches with competitive baselines on ImageNet OOD data, including an aggregation ablation.

  • ResNet-101 classifiers replace the final layer with two fully connected layers and are fine-tuned with logistic sigmoid outputs for each multi-label dataset.
  • 87.73% mAP is reported for PASCAL-VOC, 72.77% for MS-COCO, and 61.47% for NUS-WIDE.
  • Table 4 compares energy-based approaches with competitive baselines using a subset of ImageNet classes as OOD test data.
  • Table 5 ablates aggregation methods for prior approaches, while summation is not applicable to tree-based or KNN-based methods such as LOF and Isolation Forest.

B Evaluation on different OOD test data

The evaluation extends beyond ImageNet by testing on the Textures OOD dataset and reporting results in Tables 6 and 7.

  • The experiments evaluate OOD detection on Textures in addition to ImageNet.
  • Results for the Textures OOD test dataset are reported in Table 6.
  • Additional Textures results are reported in Table 7.

C Baseline Methods

The baselines evaluate multi-label OOD uncertainty using label-wise predictions or feature distances, generally aggregated by the maximum label score. ODIN and Mahalanobis are adapted to multi-label classification and tuned using constructed validation OOD data.

  • Each label is predicted independently with a binary logistic classifier.
  • The reported baseline comparisons include percentage-valued results where larger or smaller values are preferred depending on the metric.Table 6 marks metric direction with upward and downward arrows, while Table 7 studies aggregation methods.
  • The baseline methods use maximum aggregation across label-wise OOD scores.Sum aggregation is unavailable for tree-based and KNN-based approaches such as LOF and Isolation Forest.
  • ODIN adapts to multi-label classification by using the maximum calibrated label-wise prediction and perturbing the input according to its binary cross-entropy loss.The selected label is the one with the largest output.
  • Mahalanobis uses feature embeddings with label-conditional means and a covariance matrix to compute distance-based scores.
  • ODIN and Mahalanobis hyperparameters are selected on validation data containing synthetic noise and corrupted in-distribution images.The validation set includes Gaussian noise, uniform noise, arithmetic and geometric image means, and randomly permuted image patches.

D Ablation Study: JointEnergy with Top Labels

JointEnergy can be applied using only the top-k label predictions in extreme multilabel settings. This approximation is supported when omitted labels have small outputs, especially negative logits, making their energy contributions near zero.

  • JointEnergy generalizes to using only the top-k predictions when the number of labels is very large.
  • When omitted labels have relatively small outputs, their label-wise energies are approximately zero and have little effect on the JointEnergy score.The passage gives E_yi = −log(1 + e^fyi(x)) ≈ 0 when logits are especially negative.
  • The top-k ablation uses k values of 3 for MS-COCO, 2 for PASCAL-VOC, and 2 for NUS-WIDE.These values are estimated as the average number of labels per training image.
Loading 2109.14162v2…