PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantically Coherent Out-of-Distribution Detection

Jingkang Yang, Haoqi Wang, Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, Ziwei Liu

arXiv:2108.11941v1cs.CV

TL;DR

The paper addresses OOD benchmarks that split data by source rather than semantic meaning, creating unrealistic evaluations. It redesigns them as SC-OOD and proposes UDG, which exploits realistic unlabeled mixtures; the approach achieves state-of-the-art SC-OOD performance.

  • Problem

    Existing OOD benchmarks split ID and OOD data by dataset source, despite semantically matching samples, limiting realistic evaluation of OOD detection.

  • Method

    UDG jointly models labeled data and realistic unlabeled ID/OOD mixtures through unsupervised clustering and group-based ID filtering.

  • Results

    The proposed approach achieves state-of-the-art performance on SC-OOD benchmarks.

  • Takeaways & Limitations

    SC-OOD provides a more realistic and challenging setting that highlights the importance of semantic coherence and data review in OOD evaluation.

  • Takeaways & Limitations

    The method assumes the unlabeled set may mix ID and OOD samples, making the ideal assumption of purely OOD unlabeled data inaccurate.

Abstract

from arXiv · show

Current out-of-distribution (OOD) detection benchmarks are commonly built by defining one dataset as in-distribution (ID) and all others as OOD. However, these benchmarks unfortunately introduce some unwanted and impractical goals, e.g., to perfectly distinguish CIFAR dogs from ImageNet dogs, even though they have the same semantics and negligible covariate shifts. These unrealistic goals will result in an extremely narrow range of model capabilities, greatly limiting their use in real applications. To overcome these drawbacks, we re-design the benchmarks and propose the semantically coherent out-of-distribution detection (SC-OOD). On the SC-OOD benchmarks, existing methods suffer from large performance degradation, suggesting that they are extremely sensitive to low-level discrepancy between data sources while ignoring their inherent semantics. To develop an effective SC-OOD detection approach, we leverage an external unlabeled set and design a concise framework featured by unsupervised dual grouping (UDG) for the joint modeling of ID and OOD data. The proposed UDG can not only enrich the semantic knowledge of the model by exploiting unlabeled data in an unsupervised manner, but also distinguish ID/OOD samples to enhance ID classification and OOD detection tasks simultaneously. Extensive experiments demonstrate that our approach achieves state-of-the-art performance on SC-OOD benchmarks. Code and benchmarks are provided on our project page: https://jingkang50.github.io/projects/scood.

1. Introduction

The paper argues that dataset-based OOD benchmarks can mislabel semantically matching samples as OOD, then introduces SC-OOD benchmarks and UDG to model realistic unlabeled ID/OOD mixtures.

  • OOD detection seeks to reject covariate- or semantic-shifted test samples so models trained on ID data avoid unreliable predictions.
  • Existing methods achieve nearly perfect DD-OOD results but suffer drastic degradation on SC-OOD, indicating that the problem remains unresolved.
  • Around 15% of Tiny-ImageNet test samples share semantics with CIFAR-10 ID categories, exposing an irrationality in dataset-based OOD splitting.
  • SC-OOD reorganizes ID and OOD sets by semantics while ignoring negligible covariate shifts, making the ID set semantically coherent and distinct from OOD.
  • UDG jointly models labeled and unlabeled data, using unsupervised clustering to enrich semantic representations and dynamically separate ID from OOD samples.
  • The proposed framework uses realistic unlabeled data containing unavoidable ID samples and achieves state-of-the-art performance on SC-OOD benchmarks.

2. Related Works

Related work covers softmax-based, generative, ensemble, and external-data approaches to OOD detection, while identifying limitations in purified OOD data and holistic treatment of unlabeled semantics.

  • OOD methods commonly calibrate softmax outputs using temperature scaling, generative models, or ensembles.
  • Realistic external datasets inevitably contain ID samples that are expensive to filter, challenging methods that assume purified OOD data.
  • Previous methods treat external OOD data holistically and neglect its internal semantic information.
  • The paper integrates deep clustering into UDG to learn representations from labeled and unlabeled data while enabling cluster-wise OOD probability estimation.

3. Our Approach

The approach jointly models labeled ID data and a realistic, potentially contaminated unlabeled set using unsupervised dual grouping (UDG). UDG filters likely ID samples into classifier training while using grouping to enrich semantic representations and support OOD rejection.

  • Problem formulation: The pipeline trains an image classifier on labeled data and an unlabeled set to classify ID samples and reject OOD samples.The unlabeled set is introduced alongside the labeled set, with the goal of correctly classifying T I and rejecting T O.
  • Unsupervised dual grouping: UDG groups labeled and unlabeled samples together, then uses group membership to filter likely ID samples from the unlabeled set.The grouping process is applied repeatedly, and ID filtering redirects samples in sufficiently ID-dominated groups to the labeled set.
  • Main task: The classification branch uses cross-entropy on labeled ID samples and entropy loss to flatten predictions on unlabeled samples.The classification head is trained with labeled data, while the entropy objective encourages uniform predictions for presumed OOD inputs.
  • In-distribution filtering: ID filtering estimates group purity from the proportion of labeled samples belonging to each class and assigns filtered samples the group-majority label.A purity threshold τ determines which unlabeled samples are returned to the labeled set; the original labeled set remains available for error correction.
  • Auxiliary task: An auxiliary fully connected head performs deep clustering using group indices, exploiting unlabeled-data semantics alongside classification and entropy objectives.The auxiliary loss trains samples to predict their assigned groups, while the updated labeled and unlabeled sets modify the main losses during training.

4. SC-OOD Benchmarks

The SC-OOD benchmarks reorganize samples from CIFAR-10 and CIFAR-100 together with five external datasets according to semantic overlap. They evaluate both ID classification and OOD detection with multiple complementary metrics.

  • Benchmark construction: SC-OOD benchmarks use CIFAR-10 or CIFAR-100 as ID datasets and re-split Texture, SVHN, Tiny-ImageNet, LSUN, and Places365 by sample semantics.Images from semantically coherent classes in nominally OOD datasets are marked as ID rather than assigned solely by dataset origin.
  • CIFAR-10 benchmark: For CIFAR-10, Tiny-ImageNet contributes 1,207 ID images and 8,793 OOD images after semantic re-splitting.The benchmark also treats Texture and SVHN entirely as OOD, while CIFAR-100 classes are disjoint from CIFAR-10.
  • CIFAR-100 benchmark: For CIFAR-100, Tiny-ImageNet contributes 2,502 ID images and 7,498 OOD images, while LSUN contributes 2,429 ID images and 7,571 OOD images.Places365 is likewise divided into 2,727 ID and 33,773 OOD images in the listed benchmark construction.
  • Evaluation: The evaluation reports four metric types covering both ID classification and OOD detection.The listed metrics are FPR95, AUROC, AUPR, and CCR@FPRn.
  • Evaluation: Lower FPR95 indicates better performance, whereas higher AUROC, AUPR, and CCR@FPRn values indicate better performance.AUPR denotes AUPR-Out in the benchmark tables, with OOD treated as the positive class.

5. Experiments

Ablations show that unlabeled-data usage, in-distribution filtering, and unsupervised dual grouping jointly improve SC-OOD detection, while UDG outperforms prior methods on both benchmarks. Further analyses find that UDG is robust with sufficiently many clusters and more stable than sample-based filtering alternatives.

  • Ablation Summary: OE loss and IDF are most effective for OOD detection, while auxiliary deep clustering further improves performance.The ablation study also reports basic CIFAR-10 classification accuracy as ACC.
  • Effectiveness of Unlabeled Data: 7.74% FPR95 improvement results from adding realistic unlabeled data, although CIFAR-10 classification accuracy falls from 94.94% to 91.87%.The mixed unlabeled set enlarges ID/OOD prediction-score discrepancy but introduces unlabeled ID samples into the OE loss.
  • Effectiveness of Unsupervised Dual Grouping: 10.4% FPR95 benefit and improved classification accuracy show that IDF adds value beyond UDG without the auxiliary clustering loss.With cleaner ID and OOD sets, the auxiliary clustering task becomes beneficial in the completed framework.
  • Benchmarking Results: UDG achieves better results than previous state-of-the-art methods on both SC-OOD benchmarks and advantages on almost all OOD-detection metrics.The comparison averages metrics over six OOD datasets per benchmark; full results are provided in the appendix.
  • Influence of Cluster Numbers: K ≥500 makes results insensitive to the predefined cluster number, whereas small K obscures ID/OOD differences and weakens IDF.Large groups can mix ID and OOD samples or fail to represent individual classes cleanly.
  • The Design Choice of IDF: UDG-based group filtering achieves the best FPR95 and is more stable than threshold- or score-sorting-based sample filtering.UDG conservatively selects groups with ID purity above τ, while SORT can accumulate errors by prematurely including incorrectly filtered samples.

6. Conclusion

The paper identifies semantic misalignment in common OOD benchmarks and introduces SC-OOD benchmarks with UDG to address it.

  • SC-OOD re-designs OOD benchmarks around semantic meaning rather than data source, creating a more realistic and challenging setting.
  • UDG achieves state-of-the-art performance on SC-OOD benchmarks while using a realistic unlabeled set.
  • The authors position SC-OOD as a basis for new OOD research and emphasize the importance of reviewing benchmark data.

A. More Details on SC-OOD Benchmarking

SC-OOD removes low-level interpolation cues and re-splits data by semantics, making OOD detection more dependent on semantic distinctions.

  • Benchmark construction: SC-OOD replaces nearest interpolation with bi-linear interpolation, producing smoother OOD images that resemble ID images more closely.
  • Benchmark construction: SC-OOD also re-splits ID and OOD test sets according to semantics rather than retaining the original data-source split.
  • Benchmark construction: Table A1 tracks four transitions from CIFAR-10 plus Tiny-ImageNet DD-OOD to SC-OOD using validation data, bi-linear interpolation, and semantic re-splitting.
  • Benchmark effects: Semantic re-splitting further reduces existing-model performance, whereas UDG shows the smallest decrease and better overall scores on both metrics.
  • Benchmark effects: UDG’s heatmaps indicate greater concentration on semantic image regions than ODIN, which can be distracted by irrelevant visual content.

B. Visual Heatmap Comparison

The heatmap comparison indicates that UDG attends more consistently to semantic regions than ODIN during prediction.

  • The visualization compares ODIN in the upper portion with UDG in the lower portion across images from Figure 3.
  • UDG concentrates more on semantic image areas, while ODIN can focus on irrelevant regions and become distracted by objects such as a green bucket.

C. Visualization of the Proposed IDF

The IDF visualization shows that UDG can filter high-purity ID groups while retaining some lower-confidence ID samples and mitigating overconfident errors.

  • High-purity ID groups primarily contain unlabeled samples belonging to their corresponding ID categories.
  • UDG’s group-based filtering can include ID samples with individual confidence around 0.3 in the bird cluster.
  • The filtering strategy can prevent mistakes caused by neural networks assigning high confidence to incorrect images.
  • The visualization supports the reliability of the proposed IDF method.

D. Detailed Results and More Architectures

UDG maintains advantages across nearly all metrics and architectures, including difficult and easy benchmark datasets. Its strong mean performance is especially associated with gains on Tiny-ImageNet, LSUN, and Places365.

  • More Architectures: UDG has advantages on almost all metrics across WideResNet-28 experiments.The results generally follow the same trend as the ResNet-18 architecture.
  • Detailed Results: Tiny-ImageNet, LSUN, and Places365 contribute substantially to UDG’s good mean OOD-detection performance.These datasets are considered difficult because many objects have similar but different semantics.
  • Detailed Results: UDG also achieves good results on the easier Texture and SVHN datasets.

E. Valuable Comments from Rebuttal

The rebuttal clarifies that classic OOD detection targets conservative handling of covariate and semantic shifts. It also identifies an impractical benchmark goal: separating semantically identical images across datasets despite negligible covariate shift.

  • On Motivation of SC-OOD: Classic OOD detection trains a conservative model to distinguish covariate shifts in p(x) or semantic shifts in p(y).
  • On Motivation of SC-OOD: Perfectly distinguishing CIFAR cars from ImageNet cars is described as impractical when their covariate shift is negligible.
  • On Motivation of SC-OOD: The benchmark goal is unrealistic because the CIFAR and ImageNet car examples have the same semantics despite negligible covariate shift.

F. Discussion on Drawbacks

The paper acknowledges that UDG improves OOD detection while not yet matching standard ID classification, and reports consistent gains across several appendix benchmark configurations. Further work targets stronger classification and larger, more complex datasets.

  • Discussion on Drawbacks: UDG alleviates OE’s classification decline but does not yet exceed standard ID classification performance.The authors seek better use of unlabeled data while retaining OOD detection capabilities.
  • Discussion on Drawbacks: Future analysis will examine UDG on larger datasets such as ImageNet with high-resolution images and complex semantics.
  • Appendix Results: UDG obtains consistently better OOD-detection metrics on CIFAR-10 and CIFAR-100 with ResNet-18.The appendix tables report performance details for both benchmarks.
  • Appendix Results: UDG obtains consistently better OOD-detection metrics on CIFAR-10 and CIFAR-100 with WideResNet-28.The appendix tables provide performance details for both benchmarks.
Loading 2108.11941v1…