PaperlayerView exact revision on arXiv

Source-linked AI summary

PWLR: Pairwise Witness Local Rejection for Boundary-Aware Out-of-Distribution Detection

Chengyao Jia, Ruixuan Wang

arXiv:2608.15802v1cs.CVcs.LG

TL;DR

Near-OOD samples can lie close to ID class boundaries, making whole-image scores insufficient for reliable rejection. PWLR uses offline MLLM-generated pairwise local witnesses, screened with ID-only data, and consistently improves strong vision-language baselines across OOD settings and backbones.

  • Problem

    Near-OOD samples can lie close to ID class boundaries, making them difficult to reject using only whole-image class scores.

  • Method

    PWLR uses offline MLLM-generated pairwise witness phrases, screens them with ID-only data, and verifies local evidence against rival classes alongside global plausibility.

  • Results

    PWLR consistently improves strong vision-language baselines across far-OOD, cleaner and challenging OOD, and split near-OOD benchmarks.

  • Takeaways & Limitations

    PWLR can serve as a plug-in module across different base detectors and backbones, supporting language as explicit pairwise evidence for vision-language OOD detection.

  • Takeaways & Limitations

    PWLR assumes that valid witnesses appear only in part of the object, motivating reliance on the strongest few local views.

Abstract

from arXiv · show

Out-of-distribution (OOD) detection remains challenging for image classifiers, especially when near-OOD samples lie close to in-distribution (ID) class boundaries. Recent vision-language detectors improve OOD detection through class semantics, local prompting, or LLM-generated outlier concepts, but seldom use language as explicit boundary evidence between confusing ID classes. We propose Pairwise Witness Local Rejection (PWLR), which uses an MLLM offline to describe visible local cues that favor one ID class over a specific rival class. These cue phrases are then screened with ID-only data under a frozen vision-language backbone, so that only reliable local verifiers are kept. At inference, PWLR first retains a small set of globally plausible classes, then checks whether any of them is locally supported against its most relevant rivals, and finally combines this pairwise local evidence with the global class score through calibration. Experiments on ImageNet-100 far-OOD, cleaner/challenging OOD and near-OOD benchmarks show that PWLR consistently improves strong vision-language baselines across multiple backbones. Source code will be released.

1 Introduction

PWLR addresses near-OOD detection by modeling visual boundaries between confusing ID classes through directed pairwise local evidence. It uses offline MLLM-generated witnesses, ID-only screening under a frozen vision-language backbone, and calibrated combination with global candidate scores.

  • Motivation: Near-OOD samples near ID class boundaries are difficult to reject using only whole-image class scores.Such failures can cause high-confidence misclassification and severe consequences in safety-critical fields.
  • PWLR framework: PWLR reframes OOD detection as directed pairwise witness verification within each candidate class’s competitive neighborhood.The framework describes visual differences separating an ID class from a confusing rival and checks corresponding local image evidence.
  • Witness induction and screening: An offline MLLM generates fine-grained class-comparison witnesses, which ID-only samples screen into stable local evidence verifiers under a frozen VLM backbone.Witnesses are treated as rival-conditioned local hypotheses rather than free-form test-time outputs.
  • Decision rule: PWLR combines calibrated pairwise local verification with a global candidate prior, using local evidence for directed competition and the global branch for semantic plausibility.The global branch constrains retained candidates while the local branch models specific class boundaries instead of generic regional confidence.

2 Method

PWLR builds an offline, backbone-specific bank of pairwise local witnesses and a directed competition graph, then performs online OOD detection by filtering globally plausible classes, verifying them against relevant rivals, and calibrating global and local scores. Reliable witnesses are screened using ID-only data, while localized evidence is aggregated conservatively to support boundary-aware rejection.

  • Overview: PWLR has four stages: offline pairwise witness generation and graph-based candidate filtering, followed online by local verification and calibrated acceptance.The offline stages build a reusable witness bank and competition graph fitted to the selected backbone.
  • Pairwise witness generation: For each directed anchor–rival pair, an MLLM uses supporting images to generate reusable phrases describing local evidence favoring the anchor over the rival.The witness bank is built offline rather than querying an MLLM at test time; candidate phrases are ranked by occurrence, visibility, and context-risk statistics.
  • Graph construction and filtering: PWLR organizes witnesses in a directed graph whose edges encode anchor–rival comparisons and whose text representations use multiple prompt phrasings.Class prototypes are fused with text representations to score global plausibility and define semantic rival neighborhoods.
  • Graph construction and filtering: At inference, PWLR retains the top-K_g globally plausible ID classes, then verifies each against up to two relevant outgoing rivals from the witness graph.Semantic-neighbor pruning removes irrelevant rivals before pairwise verification; if none qualifies, the most similar outgoing rival is retained.
  • Pairwise local verification: Witness responses use the strongest few local views, and each retained class receives the minimum penalized directed support across its active rivals as its local verification score.TopKMean preserves localized evidence while reducing dilution from irrelevant regions; reverse-edge evidence penalizes support without becoming positive support for the class.
  • Retained-class calibrated acceptance: Calibration uses ID calibration images to map each retained class’s global score G_y(x) and local score A_y(x) through their empirical cumulative distribution functions.The resulting distributions provide the basis for combining global plausibility with pairwise local verification during acceptance.

3 Experiment

The experiments evaluate PWLR on ImageNet-100 across far-OOD, cleaner/challenging, and near-OOD benchmarks using multiple frozen vision-language backbones. PWLR achieves strong detection performance, benefits from combining global and pairwise local evidence, and remains stable across key hyperparameter ranges.

  • Datasets and setup: Experiments use ImageNet-100 with four standard far-OOD datasets and additional cleaner, realistic, and near-OOD benchmarks.SigLIP2 ViT-L/16-256 is the default frozen backbone, with PE-Core-L/14-336 and DFN2B-CLIP ViT-L/14 used to test cross-backbone generality.
  • Conventional far-OOD results: PWLR-CSP achieves 5.95% FPR95 and 98.55% AUROC, while PWLR-NegLabel achieves 6.85% FPR95 and 98.23% AUROC on conventional far-OOD.Gains are particularly clear on Places and remain consistent on SUN and Textures, while iNaturalist shows a smaller margin because strong baselines are near saturation.
  • Ablation results: 16.42% FPR95 and 96.44% AUROC result from combining witness phrases with the correct competitive neighborhood.The ablation evaluates the global prior, pairwise witness verification, reverse-edge penalty, and active rival pruning on cleaner/challenging benchmarks.
  • Sensitivity analysis: PWLR remains stable across scanned ranges of the local-global fusion weight, text-prototype fusion coefficient, retained-class set size, and local aggregation parameter.For local witness aggregation, k_p = 2 performs best in both plug-in variants because averaging too many local views can dilute localized evidence.

4 Related Work

Vision-language OOD detectors match image features with textual ID-class concepts, while newer methods exploit local or regional cues and large language models. However, existing approaches generally use local information for refinement, prompt learning, or shortcut suppression rather than explicit pairwise boundary verification under class competition.

  • OOD detection with pre-trained vision-language models: Early VLM methods detect OOD by matching image features against textual ID-class concepts and using the maximum temperature-scaled concept score.MCM treats textual ID-class embeddings as concept prototypes for OOD scoring.
  • OOD detection with pre-trained vision-language models: Subsequent methods show that local or regional cues can improve VLM-based OOD detection through score refinement, prompt learning, or background-semantic decoupling.OSPCoOp addresses shortcut learning caused by foreground-background coupling by decoupling background semantics.
  • LLM/MLLM/LVLM-assisted OOD detection: Existing local-cue approaches do not explicitly verify pairwise class boundaries under competition, motivating boundary-aware local rejection.The related work characterizes local information as supporting refinement, prompt learning, or shortcut suppression rather than pairwise boundary verification.

5 Conclusion

PWLR is proposed as an image-classification OOD detection framework that uses pairwise local evidence. Extensive evaluations show consistent improvements over strong vision-language baselines, especially under semantic-shifted and near-OOD conditions, while supporting use as a plug-in module for different base detectors.

  • Conclusion: PWLR is proposed as a framework for image-classification OOD detection.The framework is named Pairwise Witness Local Rejection.
  • Conclusion: PWLR consistently improves strong vision-language baselines across far, cleaner, challenging, and split near-OOD benchmarks.The evaluations cover multiple OOD benchmark settings.
  • Conclusion: PWLR delivers especially clear gains under semantic-shifted and near-OOD settings.These settings are identified as the cases with particularly clear improvements.
  • Conclusion: PWLR can be flexibly used as a plug-in module on top of different base detectors.The conclusion emphasizes compatibility with different detector backbones or architectures.

6 Supplementary

The supplementary material details PWLR’s family-constrained rival and object-centric witness-generation protocol, followed by ID-only screening that preserves most directed relations while producing a compact verifier bank. Additional analyses show that pairwise verification improves performance, remains robust across generation models, and is stable across repeated instantiations.

  • Witness generation: GPT-4o-mini assigns each ID class one coarse visual family, restricting rival selection to visually plausible same-family classes.GPT-5.1 then selects five visually confusable rivals from each anchor’s restricted candidate pool.
  • Witness generation: Each directed pair yields 36 raw phrases, 30 valid phrases after parsing and exclusions, and 21.78 merged witnesses on average.After merging, average visibility is 2.48 and context-risk is 1.04.
  • Witness screening: 493 of 500 directed edges and 10,575 of 10,890 witness phrases survive ID-only screening, averaging 21.45 screened witnesses per retained edge.Screening mainly removes unreliable phrases while eliminating only a small number of entire directed edges.
  • Ablations: Removing pairwise verification or shuffling rival assignments reduces performance for both NegLabel and CSP under the fixed-support-image setting.These controls support the contribution of pairwise local verification beyond retaining the same witness bank.
  • Robustness and stability: Replacing GPT-5.1 with GPT-4o or Qwen2.5-VL-7B keeps both PWLR-NegLabel and PWLR-CSP above their corresponding base detectors.Repeated-generation analyses also report tightly clustered detector results, with gains exceeding run-to-run variation.
Loading 2608.15802v1…