PaperlayerView exact revision on arXiv

Source-linked AI summary

SAM3-O2D2: Zero-Shot Object Out-of-Distribution Detection by Object Class Prompting of the SAM3-Image Model

Lucas Görnhardt, Timo Bartels, Tim Fingscheidt

arXiv:2609.08281v1cs.CV

TL;DR

Object detectors can be overconfident on unseen objects, while existing foundation-model approaches can rely on feature-space assessment or costly diffusion inference. SAM3-O2D2 prompts SAM3-image with detector-predicted classes and checks for corresponding detections. It outperforms baselines across both ID datasets, both OOD datasets, and both metrics, with lower computational cost than the current state-of-the-art method.

  • Problem

    Object detectors trained on closed sets can be overconfident on unseen objects, and image-level OOD assessment does not independently classify multiple objects in an image.

    Source: §1 · p. 1

  • Method

    SAM3-O2D2 prompts SAM3-image with detector-predicted classes and marks predictions as ID when SAM3 detects an object at the corresponding location.

    Source: §1 · p. 2

  • Results

    SAM3-O2D2 outperforms all baselines across both ID datasets, both OOD datasets, and both metrics, with lower computational cost than the current state-of-the-art method.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The method requires no retraining and can be paired with arbitrary object detectors, supporting practical deployment within the evaluated scope.

    Source: §1 · p. 2

Abstract

from arXiv · show

Object detectors have shown remarkable performance in various fields, among these medical imaging, surveillance, and autonomous driving. However, they are prone to overconfidence when encountering unseen objects in real-world deployments, causing potential safety issues. To address this, detecting out-of-distribution (OOD) objects is essential for reliable object detection. Modern approaches leverage the broad semantic knowledge of foundation models such as CLIP for post-hoc few- and zero-shot OOD detection. However, these methods typically perform OOD assessment in feature space, which can be sensitive to object detector localization errors and variations in object appearance. Moreover, the current state-of-the-art (SOTA) zero-shot method performs computationally costly diffusion in inference. In this work, for our proposed zero-shot object OOD detection method SAM3-O2D2, we employ the SAM3-image foundation model in an efficient manner. Specifically, we prompt SAM3 only with the object detector's predicted classes and compare the predictions of the object detector and SAM3. An object is in-distribution (ID), if SAM3 also detects an object at the corresponding location. If SAM3 does not detect the prompted object, this indicates a mismatch between the detector's prediction and the image content, suggesting that the object is OOD. Experimental results show that our method significantly surpasses the so-far zero-shot SOTA method. Specifically, we achieve new SOTA AuROC and FPR95 metrics over both ID datasets Pascal-VOC and BDD100K and both OOD datasets MS-COCO and OpenImages.

1. Introduction

Object OOD detection assesses each detected object independently, addressing settings where one image contains multiple objects. SAM3-O2D2 uses SAM3-image to verify detector predictions at the prediction level and is reported to outperform the prior zero-shot state of the art at lower computational complexity.

1. Introduction

  • Each detected object needs an independent ID or OOD assessment, unlike image-level OOD detection, which assigns one label to the entire image.This distinction matters in applications such as autonomous driving, where images contain multiple objects.Source: §1 · p. 1
  • Earlier object OOD methods require additional trained modules, while CLIP-based few-shot methods still fine-tune modules and RONIN uses computationally costly diffusion.The paper also identifies feature-space sensitivity to object appearance and detector localization errors as a limitation of RONIN.Source: §1 · p. 2
  • SAM3-image verifies detector predictions using predicted class prompts, marking a prediction ID when SAM3 detects the object at the same location and OOD when it does not.The paper argues prediction-level decisions are more interpretable and robust to localization errors than feature-space comparisons.Source: §1 · p. 2
  • SAM3-O2D2 outperforms RONIN across two ID datasets, two OOD datasets, and an open-world setup while requiring lower computational complexity.The paper presents it as a new state of the art in zero-shot object OOD detection.Source: §1 · p. 2

2. Related Works

Prior object OOD approaches include trained add-on modules, few-shot CLIP methods, and zero-shot RONIN. SAM3-O2D2 instead assesses predictions directly using a foundation detection model, without retraining.

2. Related Works

  • Image-level OOD detection assigns a label to an entire image, whereas object OOD detection classifies each detector-predicted object independently.Traditional object-level methods train the detector alongside additional ID/OOD modules; SAFE [40] avoids retraining the detector but still trains an MLP on the full ID dataset.Source: §2 · p. 2
  • RUNA [47] and UNI-OOD use CLIP representations for object-level OOD scores but require few-shot fine-tuning of custom modules.UNI-OOD separately represents the target object and its background.Source: §2 · p. 3
  • RONIN uses class-conditioned diffusion inpainting and CLIP similarities for zero-shot object OOD detection, but its diffusion inference is costly and its feature-space assessment can be sensitive to localization and appearance.Source: §2 · p. 3
  • SAM3-O2D2 is a post-hoc zero-shot alternative that requires no retraining across object detectors or datasets and assesses OOD at the prediction level.It requires lower computational cost than iterative diffusion-based inference.Source: §2 · p. 3

3. Methods

SAM3-O2D2 prompts SAM3-image with the object detector’s unique predicted classes, then compares their detections with the detector’s predictions. Unmatched predictions are treated as OOD.

3. Methods

  • SAM3-O2D2 uses SAM3 detections to identify OOD objects among object-detector predictions.Source: §3 · p. 3

3.1. Prompting SAM3 with Predicted Classes (Ours)

  • The method retrieves precomputed SAM3 text embeddings for the detector’s predicted classes and runs SAM3 once per unique predicted class.Thus, the number of SAM3 calls per image corresponds to the number of unique predicted classes, not the number of predicted objects.Source: §3.1 · p. 3

3.2. OOD Detection (Ours)

  • Detector boxes are grouped by predicted class and compared with SAM3 predictions of that class using an IoU threshold to assess matching.The method describes this agreement as a more robust OOD signal than feature-space similarity, which can be affected by appearance and localization errors.Source: §3.2 · pp. 3–4
  • A detector prediction is ID when SAM3 detects an object under the same class prompt at the corresponding location; otherwise it is marked OOD.The method compares detections directly at the prediction level rather than comparing feature-space similarities.Source: §3.2 · p. 4

4. Experimental Setup

The evaluation compares zero-shot and training-based object OOD methods on two ID datasets and two OOD datasets, using AuROC and FPR95. It uses DeformableDETR as the detector and SAM3-image for SAM3-O2D2.

4. Experimental Setup

  • Pascal-VOC and BDD100K [46] serve as ID datasets, with OOD performance evaluated on label-disjoint subsets of MS-COCO [21] and OpenImages.The setup uses the first 400 images of Pascal-VOC, OpenImages, and MS-COCO and the first 200 of BDD100K to balance object counts.Source: §4 · pp. 4–5
  • AuROC and FPR95 measure OOD detection performance, with true positives defined as ID objects correctly classified as ID and false positives as OOD objects incorrectly classified as ID.For SAM3-O2D2, ROC curves vary the confidence threshold while keeping the IoU threshold fixed.Source: §4 · p. 5
  • All reported methods use DeformableDETR pretrained on the respective ID dataset; SAM3-O2D2 also uses off-the-shelf SAM3-image.The baselines include RONIN, image-level zero-shot OOD approaches applied to cropped objects, and training-based VOS [6] and SIREN.Source: §4 · p. 5

5. Experimental Results and Discussion

SAM3-O2D2 achieves the strongest zero-shot object OOD detection results across both ID datasets, both OOD datasets, and both metrics, while running faster than RONIN. Sensitivity analyses find robustness to localization errors and object appearance, while threshold selection reflects a trade-off between benchmark performance and broader open-world use.

5.1. Main Results

  • SAM3-O2D2 achieves new SOTA performance across both ID datasets, both OOD datasets, and both metrics, with an overall average rank of 1.0 versus RONIN’s 2.0 and 2.6.The reported average ranks for RONIN are 2.0 on Pascal-VOC and 2.6 on BDD100K.Source: §5.1 · p. 5, Table 1 · p. 6
  • All tested IoU thresholds outperform RONIN on the reported Pascal-VOC setup, and θIoU=0.1 performs best, but the authors select 0.2 for broader open-world generalization.A very low threshold could accept insufficiently localized matches and increase false positives when ID and OOD objects co-occur in an image.Source: §5.1 · p. 5, Table 2 · p. 6
  • At 0.1 s per frame, SAM3-O2D2 runs faster than RONIN’s fastest configuration at 0.48 s while achieving better detection performance than its strongest configuration.The comparison measures method runtime on an NVIDIA A100 40GB, excluding the shared object detector.Source: §5.1 · pp. 5–6, Figure 3 · p. 7
  • Most OOD images contain R=0 or R=1 unique predicted classes, and limiting SAM3-O2D2 to R≤3 still outperforms RONIN in all cases.The detector is not called in 37...76% of frames and is called once in 19...53%; when more than three classes are predicted, the method prompts the three with lowest average detector confidence.Source: §5.1 · pp. 6–7, Figure 4 · p. 7

5.2. Sensitivity Analysis Ablation

  • Under simulated bounding-box localization errors, SAM3-O2D2 remains robust across evaluated strengths, while RONIN’s FPR rises by 6.8% absolute at ϵ=0.5.The comparison uses Pascal-VOC as ID and MS-COCO as OOD.Source: Table 4 · p. 8, §5.2 · p. 8
  • Both methods perform worse in low lighting, but SAM3-O2D2 scores 90.76% AuROC on small objects, compared with RONIN’s near-chance 51.09%.The appearance analysis evaluates lighting and object-size categories separately.Source: Table 5 · p. 8, §5.2 · p. 8

5.3. Qualitative Results and Failure Analysis

  • Qualitative success cases and failure analysis are provided in Appendix H.The supplied passage does not specify the cases or failure modes.Source: §5.3 · p. 8

6. Conclusions

SAM3-O2D2 establishes a new state of the art in zero-shot object OOD detection, outperforming all baselines across the evaluated ID and OOD datasets and metrics while using less computation than the current SOTA method.

6. Conclusions

  • SAM3-O2D2 outperforms all baselines across both ID datasets, both OOD datasets, and both metrics, with lower computational cost than the current SOTA method.The paper also reports greater robustness of prediction-level OOD assessment compared with feature-space similarity.Source: §6 · p. 8

A. AP Results of the Object Detector

The section reports the DeformableDETR detector’s average precision on Pascal-VOC and BDD100K; this detector is used to evaluate all object OOD detection methods.

A. AP Results of the Object Detector

  • DeformableDETR’s average precision is reported on Pascal-VOC and BDD100K, and the detector is used to evaluate all object OOD detection methods.The passage does not provide AP values.Source: A. AP Results of the Object Detector · p. 12

B. Results on Full Test Sets

On the full test sets, SAM3-O2D2 outperforms RONIN by a larger margin than the previously reported comparison. Lower IoU thresholds improve performance, but the authors use 0.2 to better generalize to realistic settings where ID and OOD objects co-occur.

B. Results on Full Test Sets

  • On the full test sets, SAM3-O2D2 outperforms RONIN by a larger margin than in RONIN’s official reported numbers.Evaluation uses Pascal-VOC and BDD100K as ID datasets and MS-COCO and OpenImages as OOD datasets.Source: B. Results on Full Test Sets · p. 12

C. Performance Influence of the IoU Threshold

  • Performance increases as θIoU decreases, with θIoU = 0 achieving the best results across all metrics and datasets.The analysis covers Pascal-VOC and BDD100K as ID datasets, MS-COCO and OpenImages as OOD datasets, and AuROC and FPR95.Source: C. Performance Influence of the IoU… · p. 12
  • The authors choose θIoU = 0.2 because θIoU = 0 may generalize poorly when ID and OOD objects co-occur, where weakly localized matches would raise FPR.For evaluations where ID and OOD objects cannot co-occur, they recommend also reporting results at θIoU = 0.Source: C. Performance Influence of the IoU… · p. 12

D. Evaluation in an Open-World Setup

In the open-world evaluation, SAM3-O2D2 outperforms RONIN by a larger margin in a more challenging setting where ID and OOD objects can share an image. Its computation is substantially lower than RONIN’s across the tested prompted-class counts.

D. Evaluation in an Open-World Setup

  • The open-world evaluation matches DeformableDETR predictions to ground truth at IoU > 0.3, labels same-class matches ID and other-class matches OOD, and allows both to co-occur in an image.Other predictions are ignored; the evaluation uses DeformableDETR trained on Pascal-VOC and the full MS-COCO validation set.Source: D. Evaluation in an Open-World Setup · p. 12
  • In the open-world setup, SAM3-O2D2 outperforms RONIN by a larger margin, although both methods perform worse than in the Pascal-VOC–MS-COCO setup.The authors also find that θIoU = 0.2 outperforms θIoU = 0.0 in this setting.Source: D. Evaluation in an Open-World Setup · pp. 12–13, Table 9 · p. 13

E. Comparison of Computational Complexity

  • SAM3-O2D2 requires substantially fewer TFLOPs than RONIN for R = 1, 2, and 3 prompted classes.RONIN’s computation scales linearly with prompted classes because it runs diffusion for each class; SAM3-O2D2 adds only a SAM3 detector evaluation per class.Source: E. Comparison of Computational… · p. 13, Table 10 · p. 13

F. Comparison with Few-Shot Methods

SAM3-O2D2 is competitive with few-shot UNI-OOD and, with θIoU = 0.0, achieves the best result in 7 of 8 dataset-metric combinations. UNI-OOD outperforms the zero-shot RONIN baseline across all reported datasets and metrics.

F. Comparison with Few-Shot Methods

G. Simulation of Bounding-Box Localization Error

The section simulates detector localization errors by perturbing predicted boxes with noise and sampling object-specific coordinate offsets, while keeping boxes within image boundaries.

G. Simulation of Bounding-Box Localization Error

  • A detector-predicted box is represented over image pixel positions, with a top-left corner and a height-by-width size.Source: G. Simulation of Bounding-Box… · p. 13
  • Localization errors are simulated by perturbing detector boxes with a coordinate noise factor ε ≥ 0, which determines the noise strength.The resulting object-specific uniform distributions are used to generate coordinate offsets.Source: G. Simulation of Bounding-Box… · p. 13
  • Sampled offsets perturb the bounding-box coordinates, with offsets constrained so disturbed boxes do not extend beyond the image.Source: G. Simulation of Bounding-Box… · p. 13

H. Qualitative Results and Failure Analysis

The qualitative examples show successful ID confirmation and OOD rejection, alongside failures caused by missed or misleading matches. Failure analysis attributes most false negatives at the analyzed operating point to detector errors rather than SAM3 confirmation failures.

Qualatative Results

  • Pascal-VOC examples show SAM3 confirming detector boxes for ID objects, while OOD objects without matching SAM3 boxes are correctly marked OOD.The figure presents success cases in the top and middle rows and failures in the bottom row.Source: H. Qualitative Results and Failure… · p. 14, Figure 5 · p. 15
  • A chair near an armchair is falsely marked OOD when SAM3 misses it, while a pineapple under a potted-plant prompt is incorrectly accepted as ID.Source: H. Qualitative Results and Failure… · p. 14
  • OpenImages performance may be partly affected by visible ID objects in OOD images, which the evaluation protocol expects to be marked OOD.A visible bottle is detected by SAM3 and marked ID, creating an apparent error under that protocol.Source: H. Qualitative Results and Failure… · p. 14

Failure Analysis

Loading 2609.08281v1…