PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-modal Proxy Evolving for OOD Detection with Vision-Language Models

Hao Tang, Yu Liu, Shuanglin Yan, Fei Shen, Shengfeng He, Jing Qin

arXiv:2601.08476v2cs.CVcs.MM

TL;DR

Zero-shot OOD detection needs proxy signals that remain reliable under distribution shift despite lacking labeled negatives. CoEvo adapts textual and visual proxies jointly at test time, achieving state-of-the-art benchmark performance, including improved AUROC and reduced FPR95 on ImageNet.

  • Problem

    Fixed textual proxies sparsely cover semantics beyond ID classes and remain static as visual features drift, causing cross-modal misalignment and unstable zero-shot OOD predictions.

  • Method

    CoEvo maintains textual and visual proxy caches and performs bidirectional, sample-conditioned test-time co-evolution without updating backbone parameters or using labeled OOD samples.

  • Results

    CoEvo achieves state-of-the-art performance on standard benchmarks, improving ImageNet AUROC by 1.33% and reducing FPR95 by 45.98% over strong negative-label baselines.

  • Takeaways & Limitations

    Combining textual and visual evolution yields complementary benefits for robust OOD detection, reaching an average FPR95 of 10.22% and AUROC of 97.95%.

  • Takeaways & Limitations

    CoEvo relies on a well-trained vision-language backbone, whose embedding space may inadequately capture task-relevant semantics under major shifts from pretraining domains.

Abstract

from arXiv · show

Reliable zero-shot detection of out-of-distribution (OOD) inputs is critical for deploying vision-language models in open-world settings. However, the lack of labeled negatives in zero-shot OOD detection necessitates proxy signals that remain effective under distribution shift. Existing negative-label methods rely on a fixed set of textual proxies, which (i) sparsely sample the semantic space beyond in-distribution (ID) classes and (ii) remain static while only visual features drift, leading to cross-modal misalignment and unstable predictions. In this paper, we propose CoEvo, a training- and annotation-free test-time framework that performs bidirectional, sample-conditioned adaptation of both textual and visual proxies. Specifically, CoEvo introduces a proxy-aligned co-evolution mechanism to maintain two evolving proxy caches, which dynamically mines contextual textual negatives guided by test images and iteratively refines visual proxies, progressively realigning cross-modal similarities and enlarging local OOD margins. Finally, we dynamically re-weight the contributions of dual-modal proxies to obtain a calibrated OOD score that is robust to distribution shift. Extensive experiments on standard benchmarks demonstrate that CoEvo achieves state-of-the-art performance, improving AUROC by 1.33% and reducing FPR95 by 45.98% on ImageNet-1K compared to strong negative-label baselines.

Introduction

Zero-shot OOD detection must identify unseen inputs despite overconfident predictions and limited labeled negatives. CoEvo addresses static negative-label proxies through sample-conditioned co-evolution of textual and visual proxies, improving cross-modal alignment and benchmark performance.

  • Motivation: OOD detection rejects unseen inputs that can trigger overconfident, incorrect predictions in safety-critical applications.The motivation includes healthcare and autonomous driving settings.
  • Existing Methods: Negative-label methods use semantically dissimilar textual labels as “not-ID” proxies alongside ID labels for zero-shot OOD classification.Examples include WordNet negatives in NegLabel and adjective-augmented labels in CSP.
  • Limitations: Static negative sets sparsely cover semantic space and become misaligned with shifting visual features, destabilizing cross-modal similarity geometry and decision thresholds.These limitations leave sample-specific negatives unrepresented and preserve preset textual priors during distribution shift.
  • Proposed Framework: CoEvo performs bidirectional, sample-conditioned adaptation without backbone updates or labeled OOD samples, using visual cues to mine textual negatives and textual updates to refine visual boundaries.The mechanism forms a closed loop across modalities.
  • Proposed Framework: CoEvo maintains online positive/negative proxy caches, iteratively aligns cross-modal similarities, enlarges local OOD margins, and adaptively re-weights dual-modal scores.The framework is training- and annotation-free at test time.
  • Results: 1.33% AUROC improvement and 45.98% FPR95 reduction were achieved on ImageNet over strong negative-label baselines.The reported comparison is against the best-performing negative-label baselines.

Related Work

Visual OOD detection methods primarily use image features and include score-based, distance-based, and generative-based approaches.

  • Visual OOD Detection: Visual OOD detection methods are broadly categorized as score-based, distance-based, or generative-based approaches.Score-based methods are described as particularly prominent because they introduce scoring mechanisms to discriminate ID and OOD samples.
  • Visual OOD Detection: Score-based methods use scoring mechanisms to distinguish in-distribution from out-of-distribution samples.

Methodology

CoEvo performs test-time co-evolution of textual and visual proxy caches for zero-shot OOD detection, addressing uncovered negative space and cross-modal misalignment under distribution shift. It mines contextual textual negatives, refines visual proxies from high-confidence samples, and combines modality scores after proxy evolution.

  • CoEvo formulates zero-shot OOD detection as distinguishing ID classes from unseen classes using vision-language representations without training samples or prompt tuning.
  • Static textual proxies leave semantic negative space uncovered and become misaligned with drifting visual features under distribution shift.
  • Textual Proxy Cache: Textual proxies use fixed positive ID embeddings and evolving negative embeddings, with negative candidates retrieved from a broad semantic corpus conditioned on each test image.
  • Visual Proxy Cache: Visual proxies complement textual semantics by storing high-confidence ID and OOD image features in class-aligned queues, with priority updates retaining representative distribution shifts.
  • Multi-modal Scoring: CoEvo combines textual and visual OOD scores with modality weighting, then recomputes post-evolution scores so refined visual proxies can influence final detection.
  • Proxy-Aligned Co-Evolution Mechanism: Proxy-aligned co-evolution iteratively updates both modalities to realign similarities, preserve stable ID anchors, and enlarge local OOD margins without changing backbone weights.

Experiment

CoEvo is evaluated across ImageNet-1K, OpenOOD, ablations, sensitivity analyses, and imbalance settings using online proxy evolution for zero-shot OOD detection. Results show strong benchmark performance, complementary textual and visual evolution, adaptive fusion, and robustness across data conditions.

  • Evaluation Protocol: CoEvo initializes textual and visual proxy queues from ID labels and negative text corpora, then updates them online for each test sample.The procedure computes textual, visual, and fused preliminary scores, applies an adaptive threshold, updates confident ID/OOD queues, and returns post-evolution decisions.
  • Evaluation Protocol: Experiments use CLIP ViT-B/16 with ImageNet-1K as ID data and multiple OOD datasets, including iNaturalist, SUN, Places, Textures, and OpenOOD settings.Evaluation reports FPR95, AUROC, and ID ACC across standard and near-/far-OOD conditions.
  • Main Results: 10.22% average FPR95 and 97.95% AUROC are achieved by CoEvoNegLabel on ImageNet-1K, with a 45.98% margin over the strongest baseline.The comparison includes both training-based and training-free methods.
  • Main Results: 14.47% average FPR95 and 96.70% AUROC are obtained in Far-OOD OpenOOD settings, while Near-OOD performance reaches 66.88% FPR95 and 74.65% AUROC.CoEvoCSP also reports 67.36% average ID ACC, surpassing competing training-free methods.
  • Ablation and Sensitivity: Combining textual and visual evolution reduces average FPR95 to 10.22% and raises AUROC to 97.95%, outperforming either evolution component alone.The ablation attributes complementary gains to textual semantic alignment and visual adaptation to test-distribution shifts.
  • Ablation and Sensitivity: Adaptive score fusion performs best around λ = 0.8, while textual-negative retrieval performs best at N = 5 before larger values saturate or decline.The fusion analysis links changing weights to evolving modality reliability; larger N can add redundancy or weakly aligned candidates.
  • Robustness Analysis: CoEvo consistently outperforms baselines across ID:OOD imbalance ratios, including the extreme 100:1 setting with only 100 OOD samples.The imbalance experiments use ImageNet-1K as ID data and SUN as OOD data.

Conclusion

CoEvo enables bidirectional, sample-conditioned adaptation across visual and textual modalities through evolving proxy caches and calibrated score fusion. Experiments report consistent improvements over negative-label baselines, while the appendix documents supporting technical and experimental analyses.

  • Conclusion: CoEvo adapts visual and textual modalities bidirectionally at test time through proxy-aligned co-evolution without updating backbone parameters.It iteratively realigns cross-modal similarity under distribution shifts and uses multi-modal score evolution for calibrated OOD scores.
  • Conclusion: CoEvo maintains modality-specific proxy caches and refines them iteratively to support calibrated OOD scoring under distribution shift.
  • Conclusion: Experiments on standard benchmarks demonstrate consistent improvements over negative-label baselines.
  • Appendix: The appendix covers adaptive thresholding, extended experiments, related work, and limitations of CoEvo.

Appendix A: Adaptive Threshold δ

The adaptive threshold selects a decision boundary from incoming multi-modal OOD scores by minimizing intra-class variance. A confidence margin then excludes ambiguous samples from proxy updates to improve cache reliability.

  • Adaptive threshold: Fixed thresholds can be suboptimal because ID and OOD score distributions vary across test environments.
  • Adaptive threshold: The adaptive threshold δ minimizes intra-class variance of incoming multi-modal OOD scores.This data-driven estimate is updated from the score distribution of test data.
  • Adaptive threshold: The estimated threshold can be updated incrementally over a sliding window without labeled OOD data.
  • Confidence filtering: The confidence margin γ filters samples near δ* so ambiguous examples are excluded from proxy updates.
  • Confidence filtering: Selective caching retains only high-confidence samples, preventing noisy or unstable evidence from propagating through co-evolution.

Anaylsis of CLIP Networks

CoEvoCSP remains effective across several CLIP architectures, with ViT-B/16 achieving the best overall reported results. Queue-length analysis indicates that moderate visual-proxy memory balances diversity against stale-sample noise.

  • CLIP backbone comparison: ViT-B/16 achieves the best overall results, with 97.85% average AUROC and 10.94% FPR95.
  • CLIP backbone comparison: ViT-B/32 records the lowest FPR95, 10.44%, on the Textures dataset.The passage suggests larger patch sizes may benefit fine-grained texture discrimination.
  • Queue length: Increasing queue length L initially improves AUROC by retaining a richer visual-proxy memory.
  • Queue length: Queue length beyond a moderate value causes gains to saturate and decline slightly as stale or redundant samples accumulate.The experiments set L = 10 as a balance between diversity and stability.

Adaptive Threshold vs. Fixed Threshold

The adaptive threshold outperforms fixed thresholds on ImageNet-1K for both FPR95 and AUROC, while confidence-margin filtering improves discrimination by excluding ambiguous updates. These mechanisms reduce tuning needs and stabilize proxy evolution.

  • Adaptive threshold: The adaptive threshold consistently outperforms fixed thresholds for both FPR95 and AUROC on ImageNet-1K.Fixed thresholds tested δ ∈ {0.1, 0.2, ..., 0.9}; the adaptive threshold is used in all reported experiments.
  • Adaptive threshold: Adaptive thresholding removes exhaustive threshold tuning across different ID datasets.This improves the stated applicability of CoEvo in open-world scenarios.
  • Confidence margin: Larger confidence margins γ yield higher AUROC and lower FPR95 on ImageNet-1K by filtering ambiguous samples near the threshold.Benefits saturate beyond a moderate margin, indicating a trade-off between proxy quality and retained samples.

Complexity Analyses

CoEvo is lightweight because it adds no learnable parameters or retraining, while proxy-cache maintenance introduces moderate inference overhead. Despite this cost, it improves detection efficiency–accuracy trade-offs over comparable methods.

  • Inference Complexity: CoEvo adds no learnable parameters and requires no model retraining, making it deployment-friendly.The method is evaluated on ImageNet-1K with batch size 128 using an NVIDIA RTX 3090 GPU.
  • Detection Performance: FPR95 decreases by 15.03% and 10.39% compared with NegLabel and another representative baseline, respectively.
  • Efficiency–Accuracy Trade-off: CoEvo achieves FPR95 of 10.22% versus AdaNeg’s 17.10% at comparable inference speeds of 408 versus 504 FPS.The comparison highlights CoEvo’s reported efficiency–accuracy balance.
  • Inference Complexity: Maintaining and updating dual-modal proxy caches causes moderate computational overhead during inference.The cost is reported as justified by performance gains, particularly on large-scale benchmarks.

Impact of Test Set Size

CoEvo remains effective across test streams ranging from 90 to 90K samples, consistently outperforming baselines and adapting from very limited unlabeled data. The smallest-sample regime is partly simplified by the relationship between sample count and ImageNet-1K’s label space.

  • Test Set Size: CoEvo consistently outperforms all baselines across test-set sizes from 90 to 90K samples.The experiment fixes the ID:OOD ratio at 5:4 using ImageNet as ID and SUN as OOD.
  • Low-Data Regime: At 90 samples, CoEvoNegLabel achieves FPR95 of 4.00% versus NegLabel’s 24.00%.
  • Low-Data Regime: At 90 samples, CoEvoCSP achieves FPR95 of 4.00% versus CSP’s 8.00%.
  • Adaptation: With limited unlabeled data, the proxy-aligned co-evolution mechanism rapidly adapts without prior tuning or auxiliary training signals.
  • Interpretation: The exceptionally low FPR95 at 90 samples is partly explained by an underdetermined task relative to ImageNet-1K’s 1000-class label space.
  • Interpretation: CoEvo’s advantage across data scales supports robustness and test-time adaptability in both low-data and large-scale deployment settings.

Appendix D: Limitations

The authors identify four limitations: dependence on diverse textual corpora and a suitable vision-language backbone, heuristic proxy updates, and moderate inference overhead. These constraints motivate more robust and efficient future designs.

  • Data and Language Resources: CoEvo assumes a large-scale, semantically diverse textual corpus for negative proxy mining.Proxy evolution may degrade when such corpora are unavailable or domain-specific language is underrepresented.
  • Backbone Dependence: CoEvo relies on a well-trained vision-language backbone whose embedding space may miss task-relevant semantics under major pretraining-domain shifts.The limitation is especially relevant to domains such as medical imaging and remote sensing.
  • Update Robustness: The confidence-driven proxy update mechanism may be suboptimal in highly noisy or adversarial test-time environments.The authors suggest theoretically grounded or uncertainty-aware update policies as a possible improvement.
  • Computational Cost: Maintaining and updating proxy queues incurs moderate computational overhead during inference.The authors note that this cost is amortized over the test stream.
  • Future Work: The authors leave these limitations as promising directions for future work.
Loading 2601.08476v2…