Source-linked AI summary
Zoom In, Reason Out: Efficient Far-field Anomaly Detection in Expressway Surveillance Videos via Focused VLM Reasoning Guided by Bayesian Inference
Xiaowei Mao, Bowen Sui, Weijie Zhang, Yawen Yang, Shengnan Guo, Shilong Zhao, Jiaqi Lin, Tingrui Wu, Youfang Lin, Huaiyu Wan
TL;DR
Far-field expressway anomalies are difficult to localize and interpret because small distant regions are diluted in full-frame VLM inputs, while continuous inference is computationally expensive. VIBES uses online kinematics-guided Bayesian inference to trigger focused spatiotemporal VLM reasoning. It improves anomaly detection and semantic interpretation while maintaining real-time efficiency across diverse expressway conditions.
Problem
Far-field expressway anomalies are difficult to localize and interpret because small targets, scene variation, and full-frame visual dilution weaken reliable detection.
Method
VIBES continuously updates context-dependent normal-motion boundaries from vehicle trajectories, then sends trigger-associated frames and localized regions to the VLM.
Results
VIBES improves far-field anomaly detection and semantic interpretation while maintaining real-time efficiency across diverse expressway conditions.
Takeaways & Limitations
Focused, event-centered VLM reasoning preserves evidence for fine-grained interpretation while reducing irrelevant content and unnecessary VLM invocations.
Takeaways & Limitations
The online Bayesian estimation assumes earlier ego observations and current neighboring observations are conditionally independent given the motion parameter, with neighboring observations modeled around the same expected normal-motion value.
Abstract
from arXiv · showhide
Expressway video anomaly detection is important for traffic safety, but remains challenging across diverse scenes, particularly for far-field vehicles with subtle abnormal motion. Vision-Language Models (VLMs) provide strong semantic reasoning capabilities, yet processing full frames can dilute evidence from distant targets and introduce substantial computational overhead. To address these challenges, we propose VIBES, an asynchronous framework that uses Bayesian inference to guide focused VLM reasoning. Specifically, an online kinematics-guided Bayesian inference module continuously estimates a context-dependent normal-motion distribution from vehicle trajectories and updates its probabilistic boundaries. Deviations from these boundaries produce asynchronous triggers that localize candidate anomalies in time and space. Instead of processing continuous full-frame video, the VLM reasons only over selected frames and localized visual regions associated with the triggers, reducing irrelevant visual content and unnecessary inference. Extensive experiments show that VIBES improves far-field anomaly detection and semantic interpretation while achieving real-time processing efficiency across diverse expressway conditions.
1 Introduction
Expressway anomaly detection must localize subtle far-field motion while avoiding the visual dilution and computational cost of continuous full-frame VLM processing. VIBES addresses this with Bayesian motion triggers that select focused spatiotemporal evidence for asynchronous semantic reasoning.
- Far-field anomalies are difficult because distant vehicles occupy small, variable regions and normal motion changes with viewpoint, road geometry, and traffic conditions.
- Full-frame VLM processing can dilute small abnormal regions amid dominant normal content, causing distant abnormal motion to be overlooked.
- Continuous VLM inference increases latency, whereas sparse sampling lowers computation but risks missing short anomalous events.
- VIBES continuously estimates context-dependent normal-motion distributions from vehicle trajectories and uses boundary violations to trigger candidate anomaly localization.The triggers identify relevant time and vehicle regions before semantic reasoning.
- VIBES combines Bayesian inference with focused VLM reasoning to detect far-field anomalies while reducing irrelevant content and unnecessary VLM invocations.The framework dynamically updates normal-motion boundaries, triggers localized analysis, and reports high accuracy with real-time efficiency across diverse expressway conditions.
2 Related Work
Prior video anomaly methods model visual normality, motion, trajectories, or language-assisted understanding, but far-field expressway anomalies remain difficult to localize and interpret reliably. VIBES targets this gap by coupling motion-based localization with focused semantic reasoning.
- Existing VAD progresses from reconstruction, prediction, memory, and weak supervision toward language-aided anomaly understanding, while traffic methods exploit vehicle trajectories or predicted-motion inconsistencies.
- Far-field expressway anomalies remain challenging because small targets, tracking noise, viewpoint changes, and traffic variation weaken localized motion modeling across scenes.
- Efficient video-language methods reduce processing cost through online processing, memory, event gating, frame selection, compression, or retrieval.
- Fine-grained reasoning methods improve local visual inspection, but the reviewed approaches do not directly solve spatiotemporal localization of subtle far-field motion.
3 Preliminaries and Problem Statement
The paper formalizes expressway surveillance video as a sequence of frames with tracked vehicle trajectories and defines anomaly detection as producing an explained event from video and a task prompt.
- An expressway surveillance video is represented as an ordered sequence of RGB frames over time.
- A tracked vehicle trajectory consists of valid observation times paired with vehicle bounding boxes, which need not occur in consecutive frames.
- Given video V and prompt P, VIBES outputs an event description containing the anomaly time, event category, involved vehicles, and observed cause or consequence.
4.1 Overview
VIBES uses an asynchronous zoom-in and reason-out pipeline: continuous kinematic monitoring generates triggers, localization forms focused evidence, and the VLM describes proposed events only when needed.
- VIBES continuously estimates context-dependent normal motion and groups probability-boundary deviations into event proposals for spatiotemporal localization.
- The kinematic module runs continuously, while the VLM is invoked only for localized event proposals to generate event descriptions from focused evidence.
4.2 Kinematics-Guided Bayesian Inference
VIBES estimates context-dependent normal vehicle motion from trajectories, history, and neighboring traffic, then uses Bayesian probability boundaries to identify anomalous deviations. Its online, anomaly-specific scoring converts diverse kinematic departures into triggers for downstream analysis.
- Motion representation: VIBES tracks vehicles with a lightweight tracker and represents image-plane motion in a local Frenet frame with scale-normalized velocities.Nonconsecutive observations are handled through their time intervals, while scale normalization reduces dependence on target distance.
- Online normal-motion estimation: The model estimates normal motion from earlier vehicle history and compatible neighboring observations, combining an empirical prior with current traffic evidence.The selected scalar components include longitudinal velocity, lateral velocity, and speed magnitude, whose normal parameters may vary by vehicle and traffic context.
- Online normal-motion estimation: Bayesian updating produces a reference distribution whose expected motion and variation adapt continuously as new observations arrive.The plug-in reference distribution fixes the expected value at the posterior estimate and uses estimated motion variation to define the normal boundary.
- Bayesian surprise scoring: VIBES centers recent kinematic sequences by estimated normal motion and applies anomaly-specific statistics to capture distinct temporal deviation patterns.Equal, elapsed-time, and contrasting weights represent average deviation, accumulated lateral deviation, and changes in longitudinal motion, respectively.
- Context-aware trigger generation: Boundary-normalized Bayesian surprise scores trigger when a motion pattern exceeds its probability boundary, with the largest violation identifying the vehicle-level anomaly pattern.Scores below one remain within the boundary, whereas scores above one indicate a violation; aggregation uses the largest boundary violation across motion statistics.
4.3 Focused Vision-Language Reasoning
VIBES localizes each candidate event in time and space before constructing compact visual inputs for VLM reasoning. It combines selected keyframes, localized vehicle regions, temporal storyboards, and comparison images to support event interpretation while avoiding continuous full-frame processing.
- Spatiotemporal Localization: VIBES groups nearby triggers into event proposals and uses the largest boundary-violation trigger as each proposal’s anchor.Grouping combines temporally close triggers from the same track and spatially nearby triggers from different identifiers, reducing fragmentation from track-ID changes.
- Spatiotemporal Localization: Around each anchor, VIBES selects keyframes whose temporal context depends on the motion pattern, anomaly score, and proposal duration.Stopping and deceleration retain more pre-event context, while strong lateral or longitudinal deviations may retain more post-event frames.
- Spatiotemporal Localization: For each selected frame, VIBES retains the event and interacting vehicles and enlarges their bounding-box region using a margin tied to vehicle scale.These localized regions provide the spatial evidence used to construct the focused visual inputs.
- Focused Visual Evidence: The focused inputs combine temporally ordered crops into a storyboard and keyframes around the clearest visual or kinematic transition into a compact comparison image.For moving events, crops follow the event vehicle; for abnormal stopping, a fixed stopping-location crop preserves motion before and after the stop.
- Semantic Explanation Generation: Semantic reasoning applies the VLM to the comparison image, storyboard, and prompt to identify the event vehicle, assess anomaly evidence, and describe its type or visible consequence.The VLM is invoked only after kinematic triggering and spatiotemporal localization, and can report no definite anomaly when the selected evidence is insufficient.
5 Experiments
Experiments on three datasets show that VIBES combines strong far-field anomaly detection and semantic interpretation with real-time streaming efficiency. Ablations and case studies support motion-guided localization, online adaptation, and focused VLM reasoning as central components.
- 5.3 Performance Comparison (RQ1, RQ2): 96.49% Recall, 85.96% Event Acc., and 78.95% Detail Acc. are achieved overall, with VIBES highest in every Table 2 metric row.The results cover alert timing, event categorization, and fine-grained descriptions across annotated event windows.
- 5.3 Performance Comparison (RQ1, RQ2): VIBES reaches 100.00% Recall on TUMTraf and Accident and 94.44% on CPED, improving over KIMI by 40.00, 12.50, and 11.11 percentage points, respectively.On CPED, the method detects 34 of 36 target event windows despite substantially longer videos.
- 5.3 Performance Comparison (RQ1, RQ2): The framework first applies kinematics-guided Bayesian inference, then performs focused VLM analysis on selected frames and regions rather than continuous full frames.This asynchronous design reduces redundant VLM computation while retaining evidence for anomaly detection and explanation.
- 5.4 Computational Efficiency Analysis (RQ3): VIBES achieves 2.65× throughput on Accident and 3.13× on CPED, with event-window latencies of 3.35 s and 3.26 s.It has the highest throughput on both datasets, although LLaVA-OV and Q3-VL-I have slightly lower latency.
- 5.5 Ablation Study (RQ4): Removing Bayesian localization causes a clear performance drop, while static boundaries reduce performance under changing traffic conditions.Replacing Frenet motion with absolute image-plane motion also degrades results, supporting motion decoupling across road directions and online updating.
- 5.6 Case Study (RQ1, RQ2): Case studies show that VIBES localizes brief lateral deviations and abnormal stopping under different traffic conditions, whereas full-frame VLM reasoning misses a distant event.The localized evidence supports subsequent semantic interpretation, including a suspected collision and effects on surrounding traffic.
6 Conclusion
VIBES combines lightweight kinematics with online Bayesian inference to localize candidate anomalies and focus VLM semantic reasoning on relevant visual evidence. Experiments show improved far-field anomaly detection with real-time efficiency across diverse expressway conditions.
- VIBES combines lightweight vehicle kinematics with online Bayesian inference to update context-dependent normal-motion boundaries.
- The framework triggers focused spatiotemporal localization, after which the VLM performs semantic reasoning on localized visual evidence.
- VIBES improves far-field anomaly detection while maintaining real-time efficiency across diverse expressway conditions.