PaperlayerView exact revision on arXiv

Source-linked AI summary

Can VLMs Unlock Semantic Anomaly Detection? A Framework for Structured Reasoning

Roberto Brusnicki, David Pop, Yuan Gao, Mattia Piccinini, Johannes Betz

arXiv:2510.18034v3cs.CVcs.AIcs.RO

TL;DR

Rare semantic anomalies challenge autonomous-driving safety, while VLM prompting remains unreliable and costly. SAVANT structures VLM reasoning into scene-description extraction and multimodal evaluation; its fine-tuned 7B model achieves 90.8% recall and 93.8% accuracy.

  • Problem

    Autonomous vehicles must respond to rare events that training data cannot fully capture, but VLM prompting for anomaly detection can be unreliable and costly.

    Source: §I · p. 1

  • Method

    SAVANT decomposes scenes into semantic layers, then evaluates anomalies using the image and structured descriptions.

    Source: B. Two-Phase Anomaly Detection Pipeline · p. 3, §V · p. 7

  • Results

    90.8% recall and 93.8% accuracy: the fine-tuned 7B Qwen2.5-VL model surpasses all evaluated proprietary models while enabling cost-free local deployment.

    Source: §V · p. 8

  • Takeaways & Limitations

    SAVANT annotated 9,640 real-world driving images and offers a practical approach to data scarcity in semantic anomaly detection.

    Source: §V · p. 8

Abstract

from arXiv · show

Autonomous driving systems remain critically vulnerable to the long-tail of rare, out-of-distribution semantic anomalies. While VLMs have emerged as promising tools for perception, their application in anomaly detection remains largely restricted to prompting proprietary models - limiting reliability, reproducibility, and deployment feasibility. To address this gap, we introduce SAVANT (Semantic Anomaly Verification/Analysis Toolkit), a novel model-agnostic reasoning framework that reformulates anomaly detection as a layered semantic consistency verification. By applying SAVANT's two-phase pipeline - structured scene description extraction and multi-modal evaluation - existing VLMs improve their scores in detecting anomalous driving scenarios from input images. Our approach replaces ad hoc prompting with semantic-aware reasoning, transforming VLM-based detection into a principled decomposition across four semantic domains. We show that across a balanced set of real-world driving scenarios, applying SAVANT improves VLM's absolute recall by approximately 18.5% compared to prompting baselines. Moreover, this gain enables reliable large-scale annotation: leveraging the best proprietary model within our framework, we automatically labeled around 10,000 real-world images with high confidence. We use the resulting high-quality dataset to fine-tune a 7B open-source model (Qwen2.5-VL) to perform single-shot anomaly detection, achieving 90.8% recall and 93.8% accuracy - surpassing all models evaluated while enabling local deployment at near-zero cost. By coupling structured semantic reasoning with scalable data curation, we provide a practical solution to data scarcity in semantic anomaly detection for autonomous systems. Supplementary material: https://TUM-AVS.github.io/SAVANT/.

I. INTRODUCTION

Autonomous driving must handle rare events that expose failures of contextual understanding, while VLMs offer semantic reasoning but remain difficult to deploy reliably through prompting. SAVANT introduces structured VLM reasoning for anomaly monitoring.

I. INTRODUCTION

  • Rare, low-probability events that training data cannot fully capture remain a safety challenge for autonomous vehicles.Failures such as mistaking a full moon for a traffic light illustrate the stakes for public trust.Source: §I · p. 1
  • VLMs combine visual understanding with language reasoning, making them suited to subtle contextual inconsistencies in long-tail driving anomalies.Their multimodal pretraining supports reasoning about complex scenes and spatial configurations.Source: §I · p. 1
  • Prompting VLMs for safety-critical anomaly detection can be unreliable, inconsistent, costly, and slow, motivating SAVANT’s structured reasoning framework.The figure presents scene decomposition followed by joint anomaly evaluation, with human verification of predictions supporting annotation and model fine-tuning.Source: §I · p. 1, Figure 1 · p. 1

1) Foundation Models for Driving Intelligence:

  • End-to-end driving models raise interpretability and safety-verification concerns, while VQA scene-understanding systems remain detached from control tasks.The paper positions VLM reasoning for online safety monitoring as a response to this gap.Source: 1) Foundation Models for Driving… · p. 2

2) Semantic Anomaly Detection:

The paper focuses on contextual shifts in real driving scenes, where familiar objects can form unsafe or implausible configurations. It contrasts this runtime, multimodal approach with prior methods and proposes SAVANT, model evaluation, open-model deployment, and research resources.

2) Semantic Anomaly Detection:

  • Semantic anomaly detection targets contextual plausibility shifts, beyond statistical shifts or detecting unknown objects.Earlier methods used text-only object co-occurrence reasoning or data-centric scenario mining, with limitations including simplified simulation validation.Source: 2) Semantic Anomaly Detection · p. 2
  • The paper uses multimodal VLMs on real-world images to provide direct visual evidence, contrasting with prior approaches described as text-only, simulation-dependent, or lacking interpretability.The authors present their layered method as, to their knowledge, the first automated semantic anomaly detection approach on real driving data.Source: 2) Semantic Anomaly Detection · p. 2

B. Critical Summary

  • Current approaches are described as lacking robust, interpretable, consistent prompting or relying on black-box, text-only, or simulation-validated methods.The paper frames these limitations as barriers to reliable VLM-based anomaly detection for safety-critical driving.Source: B. Critical Summary · p. 2

C. Contributions

  • SAVANT improves VLM absolute recall by 18.5% over prompting baselines and evaluates 33 state-of-the-art VLMs across performance, cost, and efficiency.The paper also reports a 7B open-source model with 90.8% recall and 93.8% accuracy, plus an extended CODA dataset containing 9,640 annotated real-world images and other research resources.Source: C. Contributions · p. 2

II. THE SAVANT FRAMEWORK

SAVANT formulates semantic anomaly detection as a binary scene-level task: identifying contextually invalid arrangements of objects. Its layered decomposition covers four traffic-scene domains, with two domains of the referenced traffic model excluded from the current scope.

A. Layered Anomaly Formulation

  • SAVANT frames anomaly detection as deciding whether a scene’s object relationships violate expected context, rather than identifying unknown objects.The output is binary; familiar objects can form an anomalous combination, such as traffic lights being transported on a truck.Source: A. Layered Anomaly Formulation · p. 3
  • The four semantic layers are Street, Infrastructure, Movable Objects, and Environment, covering the driving surface, traffic-regulating devices, dynamic entities, and weather or visibility.Together, the layers provide broad scene coverage with minimal redundancy and support targeted anomaly identification.Source: A. Layered Anomaly Formulation · p. 3
  • The framework excludes digital information and temporal modifications from its four layers.The paper cites their absence from current anomaly datasets and poor VLM temporal understanding in prior experiments, respectively, and leaves extending the layers for future work.Source: A. Layered Anomaly Formulation · p. 3

B. Two-Phase Anomaly Detection Pipeline

SAVANT replaces direct anomaly prompting with a two-phase pipeline: structured scene-description extraction followed by multimodal evaluation. The second phase uses both the image and aggregated descriptions to assess semantic inconsistencies.

B. Two-Phase Anomaly Detection Pipeline

  • In phase one, layer-specific prompts guide the VLM to describe the scene across four semantic layers instead of directly classifying anomalies.The layer descriptions are aggregated into an interpretable intermediate scene representation for the evaluation phase.Source: B. Two-Phase Anomaly Detection Pipeline · p. 3
  • In phase two, the VLM evaluates the original image together with the aggregated scene description for semantic inconsistencies.This combines visual evidence with structured textual analysis.Source: B. Two-Phase Anomaly Detection Pipeline · p. 3
  • The evaluation proceeds through layer-wise anomaly assessment, cross-layer interaction analysis, and a final binary classification with supporting rationale.The evaluation prompt instructs the model to use both visual evidence and extracted descriptions.Source: B. Two-Phase Anomaly Detection Pipeline · p. 3

C. Fine-tuning Integration Strategy

SAVANT’s structured outputs are used to create training annotations and fine-tune compact VLMs for single-shot anomaly detection. The evaluation datasets progress from a manually annotated pilot to a human-validated test set and a larger SAVANT-labeled release.

C. Fine-tuning Integration Strategy

  • SAVANT’s high-quality outputs serve as an automated annotation engine, producing training data that captures the framework’s reasoning for fine-tuning compact VLMs.The pipeline uses the resulting data to distill structured reasoning into a single-shot model.Source: C. Fine-tuning Integration Strategy · p. 3
  • A direct image prompt removes explicit layer decomposition at inference, distilling SAVANT’s multi-phase reasoning into an efficient single-shot model.The authors describe the resulting model as suitable for real-time deployment and practical autonomous-system integration.Source: C. Fine-tuning Integration Strategy · pp. 3–4

A. Datasets

  • CODALM_small contains 100 CODA-derived images, balanced between anomalous and normal scenes, with manually annotated scene descriptions and anomaly evaluations.The pilot supports screening more than 30 VLM candidates while measuring response times and API costs.Source: A. Datasets · p. 4
  • CODALM_medium combines Gemini-generated annotations with human review: 5,078 annotations were corrected, including a balanced 1,020-image test subset, while the rest support fine-tuning.Its anomalies span one to four semantic layers; 60.9% span four, and Movable Objects anomalies are most frequent at 81.7%.Source: A. Datasets · p. 4
  • The authors release CODALM_large, a 9,640-image real-world dataset labeled with SAVANT using the best-performing VLM.The experimental evaluation spans performance, scalability across VLM architectures, and practical deployment.Source: A. Datasets · p. 4, §III · p. 4

B. Models Evaluated

The evaluation includes proprietary and open-source VLMs and compares configurations with and without SAVANT’s structured reasoning. The configurations vary in input modalities, structural components, and computational requirements.

B. Models Evaluated

  • Figure 3 compares CODALM_medium with its test split by individual semantic-layer frequency, anomaly-layer combinations, and multi-layer anomaly frequency.The caption identifies four semantic layers but does not provide numerical frequencies.Source: Figure 3 · p. 4
  • The evaluation covers commercial VLM families including Gemini, GPT, Claude, Mistral, and Qwen-VL, alongside open-source alternatives ranging from 7B to 72B parameters.The proprietary models are accessed via API and require ongoing costs and external dependencies.Source: B. Models Evaluated · p. 4

C. Evaluation Methods

  • The evaluation configurations include three baselines without layered reasoning and four structured methods using SAVANT, varying modalities, decomposition, and prompt optimization.The table summarizes input modalities, structural components, and computational requirements; *_opt configurations use DSPy MIPROv2 optimization.Source: C. Evaluation Methods · pp. 4–5, Table I · p. 5

D. Evaluation Metrics and Implementation

The evaluation reports classification performance, inference time, and token use, with particular emphasis on recall for safety-critical anomaly detection. Implementation details specify prompt structure, optimization, and annotation filtering.

  • Precision, recall, F1-score, and accuracy are reported, with recall prioritized because missed true anomalies pose greater risk than false alarms.Source: D. Evaluation Metrics and Implementation · p. 5
  • All models process images at 360p, selected as a performance-efficiency trade-off based on the resolution analysis.Source: D. Evaluation Metrics and Implementation · p. 5
  • Efficiency is assessed using average inference seconds per query and average input/output tokens per query.DSPy MIPROv2 prompt optimization uses few-shot examples where applicable, and inference runs at temperature 0 for deterministic results.Source: D. Evaluation Metrics and Implementation · p. 5
  • Phase 1 prompts elicit descriptions for specific semantic layers; Phase 2 combines the image with those descriptions for anomaly classification and rationale.Source: D. Evaluation Metrics and Implementation · p. 5
  • Qwen2.5-VL-7B-Instruct is fine-tuned with LoRA and cross-entropy loss, while generated labels undergo automated consistency checks and human review of a subset.Training uses three epochs, a learning rate of 2 × 10−4, and batch size 4; reviewers correct ambiguous cases and systematic errors.Source: D. Evaluation Metrics and Implementation · p. 5

IV. RESULTS AND ANALYSIS

SAVANT’s structured reasoning and prompt optimization improve VLM anomaly detection, while the evaluation also identifies a remaining performance gap for open-source models. The reported results compare image-only baselines with structured methods and examine resolution trade-offs.

A. Baseline Performance: Image-Only Evaluations

B. Structured Reasoning Outperforms Naive Prompting

C. The Critical Importance of Multi-Modal Evaluation

SAVANT’s multi-modal evaluation combines visual perception with structured textual reasoning and outperforms single-modality alternatives. Layered guidance also improves image-only recall for a high-capacity model.

D. Model Scale and Optimization Effectiveness

Structured reasoning and optimization benefits vary with model capacity, while fine-tuning enables a 7B open model to outperform proprietary accuracy baselines. The single-shot variant also avoids intermediate pipeline steps.

D. Model Scale and Optimization Effectiveness

E. Fine-Tuned Open Model Outperforms Proprietary Ones

F. Layer-Specific Error Analysis

Single-shot fine-tuning produces strong overall results, while layer-wise analysis finds environmental anomalies especially difficult. The conclusion highlights large-scale annotation as a response to data scarcity.

F. Layer-Specific Error Analysis

  • Fine-tuned (FT) model has the lowest failure rates, below 10% for most anomalies, particularly Street (S) and Environment (E) layer combinations; non-fine-tuned (NFT) baseline exceeds 30% for most single-layer anomalies.Source: F. Layer-Specific Error Analysis · p. 7
  • Environmental anomalies remain hardest, while multi-layer scenarios show lower failure rates; the pipeline variant has approximately 40% failures on Infrastructure-related anomalies.Source: F. Layer-Specific Error Analysis · p. 7

V. CONCLUSION

  • The fine-tuned 7B Qwen2.5-VL model reaches 90.8% recall and 93.8% accuracy, surpassing evaluated proprietary models while enabling local deployment.Source: §V · p. 8
  • SAVANT automatically annotates 9,640 real-world driving images; future work includes video-based temporal analysis and on-vehicle integration.Source: §V · p. 8
Loading 2510.18034v3…