PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Foundation Model Embedding-Based Semantic Anomaly Detection

Max Peter Ronecker, Matthew Foutter, Amine Elhafsi, Daniele Gammelli, Ihor Barakaiev, Marco Pavone, Daniel Watzenig

arXiv:2505.07998v1cs.CVcs.LG

TL;DR

Semantic anomalies combine familiar visual elements in contextually invalid ways, creating a detection challenge for autonomous systems. This paper compares runtime vision embeddings with a database of nominal scenarios, using grid-based or instance-based representations and filtering. On CARLA-simulated anomalies, the filtered instance-based method matches GPT-4o while providing precise localization.

  • Problem

    Semantic anomalies are unusual combinations of familiar elements, and existing foundation-model methods can be slow or hallucinate while faster embedding methods often lack spatial localization.

    Source: §I · p. 1

  • Method

    The framework compares runtime DINOv2 embeddings with a nominal-scenario database using grid-based patches or object-centric embeddings from segmented instances, with filtering to improve robustness.

    Source: §I · p. 2, 1) Embedding Generation · p. 3

  • Results

    On the full dataset, the instance-based method with filtering matches GPT-4o and outperforms it in semantic anomaly detection while producing sharper, more complete detections.

    Source: A. Result Analysis · p. 4

  • Takeaways & Limitations

    The results show that vision foundation-model embeddings can detect and localize semantic anomalies, with performance comparable to GPT-4o.

    Source: §V · p. 5

    The instance-based method with filtering still suffers from false positives.

    Source: A. Result Analysis · p. 4

Abstract

from arXiv · show

Semantic anomalies are contextually invalid or unusual combinations of familiar visual elements that can cause undefined behavior and failures in system-level reasoning for autonomous systems. This work explores semantic anomaly detection by leveraging the semantic priors of state-of-the-art vision foundation models, operating directly on the image. We propose a framework that compares local vision embeddings from runtime images to a database of nominal scenarios in which the autonomous system is deemed safe and performant. In this work, we consider two variants of the proposed framework: one using raw grid-based embeddings, and another leveraging instance segmentation for object-centric representations. To further improve robustness, we introduce a simple filtering mechanism to suppress false positives. Our evaluations on CARLA-simulated anomalies show that the instance-based method with filtering achieves performance comparable to GPT-4o, while providing precise anomaly localization. These results highlight the potential utility of vision embeddings from foundation models for real-time anomaly detection in autonomous systems.

I. INTRODUCTION

Semantic anomalies combine familiar elements in contextually invalid ways, exposing a gap between reliable in-distribution perception and safe system-level reasoning. The paper evaluates existing VLM detectors and proposes image-embedding methods for detecting and localizing anomalies, with filtering intended to improve performance.

I. INTRODUCTION

  • Runtime OOD detection matters because inputs unlike a model’s training distribution can lead to unsafe or unpredictable behavior.A common mitigation is to detect OOD situations and transition the system to a safe state.Source: §I · p. 1
  • Semantic anomalies arrange familiar in-distribution elements in unusual contexts, such as a stop sign on a billboard or traffic lights on a truck.Such combinations can confuse autonomous vehicles despite familiar visual components.Source: §I · p. 1, Figure 1 · p. 1
  • Foundation models can be slow and prone to hallucination, while embedding methods are faster but often provide only coarse scores without spatial localization.The paper frames efficient detection with localization as an unmet need.Source: §I · p. 1
  • The paper also evaluates existing VLM-based semantic anomaly detectors at the frame level, identifying their strengths and weaknesses.Source: §I · p. 2
  • Foundation-model embeddings offer two image-based variants that detect semantic anomalies with performance comparable to GPT-4o while enabling precise localization.The evaluation uses simulated data.Source: §I · pp. 1–2
  • A filtering technique further boosts the embedding-based framework toward GPT-4o-level results.Source: §I · p. 2

II. RELATED WORK

The related work describes vision foundation models as sources of transferable visual representations and reviews language- and embedding-based approaches to anomaly monitoring. It identifies a need for a lightweight method that can both detect and localize complex semantic anomalies.

A. Vision Foundation Models

  • CLIP, DINO, and DINOv2 learn general-purpose visual features through large-scale pretraining, with DINO using self-supervision and DINOv2 trained on curated images.CLIP learns joint image-text embeddings from 400 million image-caption pairs; DINOv2 is trained on 142 million curated images.Source: A. Vision Foundation Models · p. 2
  • Segment Anything models support automatic or promptable instance segmentation, which can combine with foundation-model features to represent objects for anomaly detection.Source: A. Vision Foundation Models · p. 2

B. Foundation Models for Anomaly Detection in Robotics

  • Prior LLM-based monitoring flags robot behaviors that deviate from human intuition, with reported alignment to human judgment in driving and manipulation tasks.Source: B. Foundation Models for Anomaly… · p. 2
  • Sinha et al. use a lightweight embedding classifier to trigger slower generative reasoning and recovery, with model-predictive control enabling safe replanning.The fast detector is reported to outperform GPT-4 in failure detection.Source: B. Foundation Models for Anomaly… · p. 2
  • S2M turns anomaly scores into segmentation masks using SAM, while AnomalyCLIP learns object-agnostic prompts for detection and segmentation without target-domain data.Source: B. Foundation Models for Anomaly… · p. 2
  • The proposed framework targets a gap left by approaches that can be computationally demanding, lack localization, or do not suit complex semantic anomalies.The Fig. 2 caption states that this framework’s structure enables detection without access to out-of-distribution data.Source: B. Foundation Models for Anomaly… · p. 2, Figure 2 · p. 2

III. METHODOLOGY

The method compares runtime vision embeddings with embeddings from nominal experiences the autonomous system has handled safely, flagging observations that differ sufficiently. It is evaluated on CARLA driving scenes containing nominal cases, semantic anomalies, and out-of-distribution objects.

III. METHODOLOGY

  • Runtime observations are flagged as semantic anomalies when their vision-embedding distance from all known nominal embeddings exceeds a defined threshold.The nominal database represents scenarios previously encountered and successfully handled by the system.Source: A. Problem Formulation · p. 2
  • Evaluation uses CARLA 0.9.15 scenes with nominal cases, semantic anomalies, and out-of-distribution objects across multiple maps, positions, and variations.Examples include stop signs on billboards, trucks carrying traffic lights, and robots.Source: A. Problem Formulation · p. 2
  • Frame-level detection is measured with F1-score, balanced accuracy, true positive rate, and false positive rate; localization uses binary ground-truth masks.Source: A. Problem Formulation · pp. 2–3
  • A localization detection counts as a true positive when mask IoU is at least 0.3; any predicted anomaly in an anomaly-free frame counts as a false positive.Source: A. Problem Formulation · p. 3

B. Proposed Approach

The proposed pipeline computes DINOv2 embeddings using either a patch grid or object-centric instances, then compares runtime patches with a cache from nominal observations. A threshold calibrated on nominal scores determines whether an observation is anomalous.

B. Proposed Approach

1) Embedding Generation:

  • Grid-based extraction uses DINOv2 patch embeddings, while object-centric extraction combines OWLv2 prompts and SAM2 masks to average the embeddings of patches within each instance.DINOv2 produces 256 embeddings per image, each representing a 14 × 14 pixel region with dimension 384.Source: 1) Embedding Generation · p. 3

2) Anomaly Detection:

  • Nominal experiences are represented by offline DINOv2 patch embeddings, and runtime observations are scored by comparing their patches with nearest patches in the nominal cache.The method uses patch-level representations for finer anomaly detection and localization.Source: 2) Anomaly Detection · p. 3
  • A runtime observation is anomalous when its score exceeds a threshold set to the α-quantile of nominal scores computed leave-one-out by experiment.Excluding embeddings from the same experiment avoids nominal bias from highly similar consecutive images.Source: 2) Anomaly Detection · p. 3

3) Filtering:

A connected-component filter removes small, noisy detections without changing anomaly scores. The evaluation compares filtered and unfiltered embedding methods with GPT-4o on frame-level metrics.

3) Filtering:

  • Small connected components are removed from the binary anomaly map using a pixel threshold, without changing anomaly scoring.The filter addresses isolated high-score patches associated with a small nominal database and instance-segmentation oversegmentation.Source: 3) Filtering · p. 3
  • The filtering step reduces noise, while more advanced filtering methods are left unexplored.Source: 3) Filtering · p. 3

IV. EVALUATION AND DISCUSSION

  • Evaluation compares GPT-4o with embedding-based methods with and without instance information, using empirically determined anomaly-score and patch-size thresholds.Frame-level binary classification metrics are reported.Source: §IV · p. 3

A. Result Analysis

The instance-based method with filtering matches GPT-4o overall and outperforms it on semantic anomalies, while the two embedding variants show different strengths. Score separation and threshold performance vary by anomaly scenario.

A. Result Analysis

  • GPT-4o has the highest F1-score and lowest FPR for OOD-objects, but often misses subtle semantic anomalies such as traffic lights on trucks.The authors note that domain shift between real-world training data and synthetic CARLA images may also affect tested performance.Source: A. Result Analysis · p. 3
  • The unfiltered embedding-based method has high TPR but high FPR, and its detections often reflect visual novelty rather than the full semantic anomaly.Uncommon elements underrepresented in the nominal database can also trigger false positives.Source: A. Result Analysis · pp. 3–4
  • Filtering reduces FPR and improves F1-score by removing isolated false detections caused by noisy patch-level scoring.Source: A. Result Analysis · p. 4
  • With filtering, the instance-based method matches GPT-4o on the full dataset and outperforms it in semantic anomaly detection.It offers a more balanced sensitivity–precision trade-off and produces sharper, more complete detections; Table I reports overall and scenario-wise frame-level metrics.Source: A. Result Analysis · p. 4, Table I · p. 4, Figure 3 · p. 4

B. Score Analysis

  • Anomalies usually receive higher scores than nominal objects, but the instance-based method performs better on semantic anomalies and the embedding-based method on OOD objects.Score distributions across scenarios are shown in Fig. 4.Source: B. Score Analysis · p. 4, Figure 4 · p. 5
  • Stop-sign anomalies remain closer to nominal scores, and threshold sweeps show that higher TPR consistently comes with more false positives.The best-performing threshold differs across scenarios, complicating selection of one threshold for all cases.Source: B. Score Analysis · pp. 4–5, Figure 5 · p. 5

V. CONCLUSION AND OUTLOOK

The paper concludes that vision foundation model embeddings can detect and localize semantic anomalies with performance comparable to GPT-4o. It identifies loss of global context as a limitation and proposes richer embeddings, adaptive scoring, or graph-based scene representations as directions for further work.

V. CONCLUSION AND OUTLOOK

  • The proposed embedding-based approaches detect and localize semantic anomalies with performance comparable to GPT-4o.The paper characterizes this work as a first step toward embedding-based semantic anomaly and OOD detection.Source: §V · p. 5
  • Future work could develop embeddings that better capture anomaly characteristics or use more adaptive scoring mechanisms, such as energy-based models.Source: §V · p. 5
  • Analyzing embeddings in isolation may lose global context needed for anomalies involving atypical object combinations; graph-based representations may support joint reasoning while preserving context.Source: §V · p. 5

APPENDIX

The appendix gives examples of evaluated anomaly scenarios and visualizes intermediate outputs from the embedding-based and instance-based pipelines.

A. Examples images of the different anomaly scenarios

  • Evaluation examples show OOD objects, traffic lights, and stop signs as the three anomaly scenarios.Source: Figure 6 · p. 7

1) Embedding-Based Anomaly Detection Visualization:

  • The embedding-based heatmap highlights the anomalous object, while thresholded classification produces a blurry mask extending beyond it.The appendix attributes the reduced spatial resolution to using 256 embeddings for the full image.Source: 1) Embedding-Based Anomaly Detection… · p. 7

2) Instance-Based Anomaly Detection Visualization:

  • The instance-based visualization shows repeated streetlight detections and overlapping boxes that hinder SAM2 segmentation.The background is grouped into a single mask.Source: 2) Instance-Based Anomaly Detection… · p. 7
  • Instance segmentation yields sharp boundaries around the anomalous boat, but the heatmap has weaker contrast and minor false positives.The passage links weaker contrast to embedding averaging in the large background mask, and false positives to imperfect detection and segmentation.Source: 2) Instance-Based Anomaly Detection… · p. 7

C. Prompt Used for ChatGPT-4o

The GPT-4o prompt casts the model as an autonomous-vehicle fault monitor, asking it to assess scene elements for safety and traffic-rule risks. It requests object-level judgments and an overall scenario classification.

C. Prompt Used for ChatGPT-4o

D. GPT-4o failures

The examples illustrate GPT-4o missing subtle semantic anomalies or assigning anomaly labels for unrelated reasons, which the authors associate with insufficient domain-specific context.

D. GPT-4o failures

  • GPT-4o can classify subtle semantic anomalies as nominal or flag them for the wrong reason.The authors suggest this partly reflects limited understanding of what defines a semantic anomaly.Source: D. GPT-4o failures · p. 8
  • GPT-4o classifies a stop sign on a billboard as nominal, reasoning that it provides a safety reminder without considering its relevance to autonomous driving.Source: D. GPT-4o failures · p. 8
  • In another case, GPT-4o labels an image anomalous because of vehicle misalignment rather than the traffic light it carries.It mentions the traffic light only to say that its color is unclear.Source: D. GPT-4o failures · p. 8
  • GPT-4o may omit a traffic light and classify its truck as nominal; the authors suggest domain shift as one possible explanation.They say driving-specific fine-tuning or targeted in-context examples could help mitigate these issues.Source: D. GPT-4o failures · p. 8

E. Examples of failures in the object detector

The object-detector examples focus on false positives from OWLv2 on synthetic CARLA images and their downstream effects on SAM2 segmentation and anomaly classification.

E. Examples of failures in the object detector

Loading 2505.07998v1…