PaperlayerView exact revision on arXiv

Source-linked AI summary

CODE: Cross-Modal Calibration and Dynamic Suppression for Open World Object Detection

Hao Xu, Zhaoning Shi, Hehe Jin, Bo Ma

arXiv:2608.27214v1cs.CV

TL;DR

Multimodal OWOD can suffer from semantic ambiguity in unidirectional text-to-vision matching and over-suppress unknowns near known-class boundaries. CODE combines visual-prototype calibration, uncertainty-guided unknown enhancement, and confidence-margin suppression at inference time. On RWD with OWL-ViT L/14, it improves Task 1 U-mAP and K-mAP over the previous state of the art.

  • Problem

    Unidirectional text-to-vision matching can be ambiguous, while rigid outlier suppression can wrongly suppress hard unknown objects near known-class decision boundaries.

  • Method

    CODE is an inference-time framework that calibrates text-derived known logits with visual prototypes, enhances unknown logits from local visual-response dispersion, and adjusts suppression using the top-1/top-2 confidence margin.

  • Results

    2.6 and 2.3 points: CODE improves Task 1 U-mAP and K-mAP over the previous state of the art with OWL-ViT L/14 on the Real-World Detection benchmark.

  • Takeaways & Limitations

    CODE improves unknown-object discovery and known-class recognition across diverse domains and backbone scales by combining visual references with uncertainty-aware inference.

  • Takeaways & Limitations

    CODE relies on dataset-specific visual prototypes whose representativeness may decrease under extreme data scarcity, long-tailed or noisy annotations, and cross-domain distribution shift.

Abstract

from arXiv · show

Open World Object Detection (OWOD) built on multimodal foundation models often suffers from semantic ambiguity caused by unidirectional text-to-vision matching, while rigid outlier penalties may over-suppress unknown objects near known-class decision boundaries. We propose CODE (Cross-Modal Calibration and Dynamic Suppression), a unified inference-time framework with three complementary components. Cross-Modal Joint Confidence Calibration injects global visual prototypes to calibrate text-driven known-class predictions. Uncertainty-Guided Universal Objectness Enhancement measures classification hesitation from local visual responses to strengthen potential unknown objects. Dynamic Outlier Suppression via Confidence Margin replaces rigid suppression with a margin-aware adjustment that preserves ambiguous out-of-distribution instances. Experiments on the Real-World Detection benchmark demonstrate that, with the OWL-ViT L/14 backbone, CODE achieves 21.7 U-mAP and 40.8 K-mAP in Task 1, surpassing the previous state of the art by 2.6 and 2.3 points, respectively.

1 Introduction

OWOD with multimodal foundation models must recognize known classes while discovering unknown objects, but text-only matching and rigid suppression can mishandle ambiguous instances. CODE addresses these issues through cross-modal calibration, uncertainty-guided unknown enhancement, and margin-aware suppression.

  • OWOD requires detecting known categories while localizing and flagging unknown objects in dynamic environments.
  • Unidirectional text-to-vision matching lacks a direct visual reference when textual attributes are insufficient or visually ambiguous.Occlusion, blur, viewpoint changes, and poor illumination can make visual attributes unreliable.
  • CODE calibrates text-driven known-class predictions using global visual prototypes through CMJCC.The prototypes provide a visual-to-visual reference for reducing semantic ambiguity.
  • UGUOE strengthens potential unknown objects by measuring classification hesitation from local visual-response variance.It targets objects in semantic vacuums or with weak and ambiguous textual activation.
  • DOSCM replaces rigid outlier penalties with confidence-margin adjustment based on the gap between the top two known-class probabilities.This protects ambiguous out-of-distribution objects from over-suppression.
  • 2.6 and 2.3 points: CODE improves Task 1 U-mAP and K-mAP over the previous state of the art with OWL-ViT L/14.Evaluation uses the Real-World Detection benchmark across Aquatic, Aerial, Game, Medical, and Surgery scenarios.

2 Related Works

Related work progresses from closed-set and unimodal OOD detection toward open-vocabulary and attribute-driven OWOD using multimodal representations. CODE instead performs lightweight inference-time calibration for attribute-driven OWOD without online language-model decoding or additional bidirectional detector training.

  • Early OWOD methods primarily use pseudolabeling or contrastive clustering to define decision boundaries.
  • Attribute-driven methods decompose abstract categories into fine-grained, class-agnostic attributes to identify unknown objects sharing characteristics with known classes.FOMO, UMB, and PASS use attributes such as shape, texture, and material.
  • Open-ended detection methods extend open-vocabulary detection through generative region-language modeling or dedicated visual-language alignment training.GenerateU and Open-Det represent these directions.
  • CODE targets attribute-driven OWOD with lightweight inference-time calibration using pre-computed textual attributes and visual prototypes.It avoids online language-model decoding and additional bidirectional detector training.
  • Unimodal OOD scores such as MSP, energy, and MaxLogit often struggle with semantic shifts in open-world environments.
  • Cross-modal OOD methods align visual features with known textual concepts, add negative prompts, or refine decisions through parameter-efficient fine-tuning.MCM, CLIPN, and PEFT-MCM exemplify these approaches.

3 Methodology

CODE is an inference-time framework that combines visual-prototype calibration, uncertainty-based unknown enhancement, and confidence-margin outlier suppression for attribute-based OWOD. Its components jointly address semantic ambiguity, weak unknown responses, and over-suppression near known-class boundaries.

  • Framework overview: CODE builds on attribute-based OWL-ViT inference, using optimized attribute embeddings and known-class visual prototypes computed offline.Attribute selection and adaptation optimize embeddings and mapping weights before inference; prototypes are cached and not updated with test images.
  • Framework overview: Unified logits combine cosine similarities between candidate-region visual embeddings and optimized attributes with a mapping matrix containing known and unknown outputs.The mapping matrix has K known-category columns and one unknown-category column.
  • Cross-Modal Joint Confidence Calibration: CMJCC injects cached global visual prototypes to calibrate text-driven known-class logits against visual-level class references.Prototype injection captures class visual distributions and compensates for gaps in textual attribute coverage.
  • Uncertainty-Guided Universal Objectness Enhancement: UGUOE estimates universal objectness from local visual-response uncertainty, boosting potential unknown objects whose visual evidence is strong but not concentrated on one known class.It addresses the weakness of maximum-attribute objectness, which can confuse background noise with ambiguous unknown targets.
  • Dynamic Outlier Suppression via Confidence Margin: DOSCM replaces rigid maximum-probability suppression with confidence-margin suppression, reducing penalties when Top-1 and Top-2 known-class probabilities indicate semantic confusion.The method aims to preserve ambiguous unknowns near decision boundaries while retaining known-class precision.

4 Experiments

CODE is evaluated across the RWD benchmark, ablations, sensitivity analyses, and additional benchmarks. Results show gains in known- and unknown-category detection, while performance remains constrained by visually overlapping categories and scarce prototype data.

  • Overall Performance: CODE achieves the best overall performance under both backbones, improving known- and unknown-category metrics across Tasks 1 and 2.With L/14, it obtains 21.7 U-mAP, 40.8 K-mAP, 43.6 PK-mAP, and 36.2 CK-mAP; with B/16, all four overall metrics also improve.
  • Analysis of Task 1: +9.1 U-mAP is CODE’s largest Task 1 gain, occurring on Surgery; U-mAP also improves on Aquatic, Aerial, and Medical.The reported gains are +3.1, +2.9, and +0.6 points, respectively.
  • Analysis of Task 1: Game declines by 2.8 U-mAP and 1.7 K-mAP, where average inter-class cosine similarity reaches 0.82.The analysis supports confusion between boundary unknowns and nearby known-category clusters.
  • Analysis of Task 2: +4.7 PK-mAP and +0.2 CK-mAP are achieved under L/14 in Task 2, which evaluates an expanded category set after joint retraining.PK-mAP gains reach +8.7 on Aquatic and +10.3 on Medical, while CK-mAP gains there are +1.3 and +6.3 points.
  • Ablation Study: Ablations show complementary module effects: CMJCC improves known discrimination, UGUOE raises unknown detection, and DOSCM recovers known performance.On Surgery, U-mAP progresses from 16.2 to 17.0 to 22.4 to 25.7, while K-mAP changes from 43.0 to 45.0 to 42.3 to 43.9.
  • Sensitivity Analysis: Prototype estimation is sensitive at 10 shots, while at least 50 samples provide substantially more stable K-mAP.On Aquatic, U-mAP reaches 24.7 at 100 shots; the experiment isolates sample-size sensitivity rather than a complete long-tailed or noisy-label setting.

5 Conclusion

CODE combines visual calibration, uncertainty-guided unknown enhancement, and margin-aware suppression to improve open-world detection across diverse domains and backbone scales. Its scope is bounded by prototype representativeness under scarce, noisy, or shifted data and by overlapping fine-grained categories.

  • 5 Conclusion: CODE combines CMJCC, UGUOE, and DOSCM in a unified inference-time framework for attribute-driven open-world object detection.The components calibrate known-class predictions, enhance potential unknowns, and protect ambiguous boundary instances.
  • 5 Conclusion: CODE improves unknown-object discovery and known-class recognition across diverse domains and backbone scales.The conclusion attributes this result to combining direct visual references with uncertainty-aware inference.
  • Limitations: CODE relies on dataset-specific visual prototypes whose representativeness may decrease under extreme scarcity, long-tailed or noisy annotations, and cross-domain distribution shift.The few-shot study isolates sample scarcity but does not cover all these conditions.
  • Limitations: Highly overlapping fine-grained categories remain challenging, particularly in the Game and Surgery scenarios.

A.4 Evaluation Tasks

The RWD protocol divides classes by frequency into known and unknown subsets and evaluates detection across two stages. The setup also uses LLM-generated class-agnostic attributes encoded as structured textual priors for unknown-object discovery.

  • A.4 Evaluation Tasks: The RWD split assigns the 50% most common classes to Known and the 50% least common classes to Unknown.
  • Task 1: Task 1 trains detectors on known classes and evaluates generalization to novel, unlabeled unknown objects.
  • Task 2: Task 2 reveals previously unknown classes, expands the known set, and jointly retrains using all samples available at that stage.
  • Attribute Construction: LLMs decompose known category labels into class-agnostic attributes that provide a semantic baseline for discovering unknown objects.Attributes are organized across dimensions such as shape, color, texture, context, and behavior, then formatted and encoded by OWL-ViT’s frozen text encoder.

B.2.1 Training and Loss Functions.

CODE trains attribute representations with a composite objective and executes CMJCC, UGUOE, and DOSCM sequentially at inference. Visual prototypes and local uncertainty support calibration and unknown-object enhancement, while margin-aware suppression protects boundary targets.

  • Partial Optimal Transport: Partial Optimal Transport relaxes equal-mass matching by transporting in-distribution mass and filtering redundant out-of-distribution attributes.
  • Inference Pipeline: Inference sequentially applies CMJCC for visual-to-visual calibration, UGUOE for boosting potential unknown regions, and DOSCM for dynamically protecting boundary targets.
  • UGUOE: UGUOE computes a hesitation weight from local visual-response variance and clamps it between predefined minimum and maximum values.
  • Prototype Extraction: Visual prototypes are extracted from retained predicted regions with at least 0.8 IoU to ground-truth boxes and used for class-wise prototype estimation.

C.1.1 CMJCC Module Ablation.

CMJCC’s candidate-specific dynamic gating is important for Task 2 performance: fixed or removed thresholds degrade calibration, especially by admitting noise or failing to adapt visual priors.

  • Ablation Design: CMJCC compares the full framework with fixed-floor gating and direct visual-similarity enhancement to isolate dynamic-threshold contributions.
  • Dynamic Gating: 35.6 to 27.6: replacing adaptive gating with a fixed threshold reduces CK-mAP on Surgery under Task 2 evaluation.The result indicates that rigid thresholds may under-activate valid visual priors or include background noise.
  • Direct Similarity Variant: 49.1 on Aquatic and 25.4 on Surgery: removing the gating threshold causes severe CK-mAP degradation across the reported settings.Without ReLU-based filtering, irrelevant local visual responses enter calibration and pollute text-driven logits.

C.1.2 UGUOE Module Ablation.

The UGUOE ablations show that universal objectness injection is essential for unknown recall, while uncertainty weighting selectively improves ambiguous-region detection and limits harm to known-class performance. Sensitivity analyses further show that calibration margins, neighborhood size, uncertainty strength, and activation thresholds govern the precision–recall trade-off.

  • Module design: UGUOE combines base universal objectness activation with uncertainty-guided weighting in Task 1 experiments using L/14 on Aquatic and Surgery.The ablation compares the full framework with variants removing objectness injection or uncertainty weighting.
  • Ablation results: U-mAP drops from 25.7 to 2.3 on Surgery and from 24.7 to 9.1 on Aquatic when universal objectness enhancement is removed.The degradation supports explicitly injecting aggregated local visual similarities for recalling unknown instances.
  • Ablation results: U-mAP decreases from 24.7 to 20.1 on Aquatic and from 25.7 to 23.9 on Surgery without uncertainty weighting, while K-mAP changes little.The variance-based weighting prevents indiscriminate boosting of regions with high local visual similarity.
  • Sensitivity analysis: A minimal neighborhood K_u=1 performs well on Aquatic but poorly on Surgery, whereas activation threshold choices trade unknown discovery against background suppression across domains.The default τ_unk=0.25 is more effective for Aquatic, while τ_unk=0.5 slightly improves unknown-tool discovery on Surgery.
  • Sensitivity analysis: Surgery reaches 28.1 U-mAP at γ=200, while performance remains relatively stable across 10≤γ≤200.Higher γ more aggressively penalizes high-certainty regions, helping identify ambiguous targets in the fine-grained medical setting.
  • Dynamic suppression: Dynamic suppression shrinks penalties for ambiguous targets, lifting hard medical instruments above the suppression threshold, but can increase unknown false positives in synthetic Game scenes.The trade-off is linked to extreme inter-class similarity in Game.

C.3.3 Qualitative Analysis on Task 1 Discovery.

Task 1 qualitative examples show CODE recovering unknown objects that PASS misses because of semantic similarity, occlusion, or rigid suppression. The visualizations also compare logit distributions and dynamic suppression behavior across datasets.

  • Recovered unknowns: CODE recovers unknown Tennis Courts that PASS misses or suppresses because they resemble known Basketball Courts.The examples are shown in the Aerial scenario.
  • Visualization evidence: Figure S1 compares known and unknown logit distributions across Aquatic, Medical, and Surgery, with CODE shifting probability mass into the region above 0.Figure S2 plots Top-1 known Softmax probability against final OOD score and contrasts rigid suppression with DOSCM’s dynamic boundary.
  • Recovered unknowns: CODE discovers unknown Scissors despite metallic texture and slender structure resembling the known Suction Tube category.UGUOE measures local visual variance while DOSCM applies a soft confidence margin.

C.3.4 Qualitative Analysis on Task 2.

Task 2 qualitative results indicate that CODE improves recall of currently known targets across diverse domains through visual-to-visual calibration. However, extreme category overlap and scale variation still produce omissions.

  • Cross-domain recall: CODE recalls more Currently Known targets across Aerial, Game, Medical, and Surgery than the PASS baseline.The comparison uses ground-truth CK annotations and evaluates predictions after joint retraining on the expanded category set.
  • Domain examples: CODE identifies more Basketball Court instances in Aerial, although high overlap with Tennis Courts still causes some omissions.The example highlights improved recall alongside a remaining confusion boundary.
  • Domain examples: CODE localizes additional Metacarpal Bones and Phalanges in Medical, but still misses the Radius.This shows improved target localization without complete recall.
  • Domain examples: CODE correctly identifies Bipolar Forceps Down in Surgery, compensating for inconspicuous tool attributes through global visual prototypes.PASS misses this category because pure text-driven logits are difficult to activate effectively.
  • Interpretation: The qualitative results support CMJCC as a visual calibration reference, while extreme inter-class similarity and scale variation remain challenges.The conclusion attributes improved known-class recall across multiple domains to aggregating and injecting global visual prototypes.

C.3.5 Qualitative Analysis on Class Confusion and Limitations.

CODE reduces semantic confusion in qualitative Task 1 cases by combining uncertainty-guided enhancement with dynamic suppression, but its adaptive boosts introduce false positives and boundary errors.

  • Class confusion: CODE isolates the unknown Stingray without a hallucinated known box, whereas PASS predicts overlapping unknown and known detections.The example illustrates confusion between novel targets and established categories.
  • Class confusion: CODE rescues the unknown Chimney from rigid misclassification as the known Storage Tank by adjusting the suppression boundary.UGUOE captures hesitation and DOSCM modifies suppression for the ambiguous target.
  • Failure cases: UGUOE can misclassify a known fish as unknown when high local visual variance resembles an ambiguous target.The same over-activation also helps recover unknown objects blending into complex backgrounds.
  • Failure cases: CMJCC can hallucinate a known Dam prediction on an unknown Expressway Toll Station by over-boosting similar background or target regions.The error arises when textural or structural similarities to known prototypes inflate known logits.
  • Limitations: The framework’s dynamic margins and uncertainty enhancements improve unknown recall but require further refinement to prevent over-boosting at extreme decision boundaries.The paper identifies this calibration issue as an important direction for future research.

D Limitations and Future Work

CODE remains limited by dataset-specific visual prototypes, incomplete evaluation of long-tailed or noisy settings, and ambiguity in visually similar domains. Future work targets richer prototype representations and dynamic, context-conditioned attribute generation.

  • Limitations: Dataset-specific visual prototypes may become less representative under substantial cross-domain distribution shift.
  • Limitations: Few-shot experiments isolate sample scarcity but do not fully evaluate long-tailed class distributions or noisy annotations.
  • Limitations: High inter-class visual similarity can cause UGUOE to over-boost ambiguous regions and produce unknown false positives in Game and Surgery datasets.
  • Limitations: A single visual-prototype centroid per category may fail to capture intra-class variation such as viewpoint or occlusion.
  • Future Work: Future work proposes multiple prototypes per category using clustering techniques such as K-Means to improve robustness to intra-class variance.
  • Future Work: Future work also considers Dynamic Attribute Generation, using Large Multimodal Models to create attributes in real time from visual context.
Loading 2608.27214v1…