PaperlayerView exact revision on arXiv

Source-linked AI summary

VESSI - VLM-Enhanced Support for Surveillance and Investigations

Saverio Cavasin, Pietro Tedeschi, Mattia Tamiazzo, Alessandro Brighente, Simone Milani, Mauro Conti

arXiv:2610.11674v1cs.CV

TL;DR

Forensic surveillance review relies heavily on manual inspection of large collections. VESSI applies prompt-based per-frame VLM analysis and CMUS; Qwen2.5-VL leads the reported CMUS ranking, whose ordering agrees with normal-video and human-reference comparisons.

  • Problem

    Forensic use of growing surveillance collections still relies heavily on slow, resource-intensive manual inspection.

    Source: §I · p. 1

  • Method

    VESSI is a modular, prompt-based per-frame surveillance pipeline that evaluates four VLMs using CMUS alongside detection coverage, time saved, detection rate, and hallucination index.

    Source: §IX · p. 10

  • Results

    1.6031 CMUS for Qwen2.5-VL (versus 1.1909 for LLaVA); average time savings ranged from 64.59% to 90.96%, and model ordering agreed across normal-video CMUS and human-reference FPR.

    Source: §IX · pp. 10–11

  • Takeaways & Limitations

    VESSI is positioned as a practical triage and benchmarking framework whose alerts guide subsequent human assessment.

    Source: §IX · p. 11

    Video-level labels do not provide exhaustive frame-level ground truth; the manual-reference subset covers 65 videos and 5,909 frames.

    Source: §VIII · p. 10

Abstract

from arXiv · show

Automated video surveillance analysis has become a critical component of intelligence infrastructures and Law Enforcement agencies. Traditional systems lack the semantic module for comprehensive situational awareness and forensic tasks, limiting their ability to interpret events meaningfully or support post-incident investigations. This slows operational insight and increases the burden on human analysts. Recent advances in Vision-Language Models (VLMs) offer promising pathways to bridge this gap. To address this, we propose VLM-Enhanced Support for Surveillance and Investigations (VESSI), a VLM-based framework designed to enhance automated video surveillance analysis through prompt-driven interrogation of video sequences where salient visual features are converted into textual descriptions. We test our framework with four state-of-the-art models. Since most datasets for this task are unlabeled, we also propose the Composite Model Utility Score (CMUS) to assess VLM performance. Experimental results show that our solution substantially improves the analysis capabilities of human operators and enhances the flexibility of automated surveillance systems. In our evaluation, the most reliable model flagged potentially relevant activity in more than 66% of the videos while reducing review time by more than 85%, offering a practical balance between selectivity and efficiency. The model ordering produced by the reference-free CMUS evaluation was reproduced by the normal-video CMUS evaluation and matched the false-positive-rate ordering obtained from 5,909 manually referenced frames. This agreement supports the operational use of the score within the evaluated setting.

I. INTRODUCTION

Surveillance footage is growing while forensic review remains manual; VLMs offer natural-language scene interpretation, but operational use faces resource and reliability concerns. VESSI combines prompt-driven frame interrogation, traceable evidence retrieval, and benchmarking of four VLMs.

I. INTRODUCTION

  • Large surveillance collections still require slow, resource-intensive manual forensic inspection, while VLM use also faces compute and output-reliability concerns.VLM outputs may be inconsistent or hallucinated on ambiguous or out-of-distribution inputs.Source: §I · p. 1
  • Existing video-understanding approaches often emphasize object detection and tracking, leaving structured querying, traceable retrieval, operational profiling, and evaluation without exhaustive frame labels under-addressed.Source: §I · p. 1

A. Contributions

  • VESSI proposes a lightweight modular architecture for single-frame surveillance interrogation, semantic prompting, and traceable evidence retrieval with VLMs.Source: A. Contributions · p. 1
  • CMUS combines detection behavior, inference efficiency, and hallucination control to compare VLMs when exhaustive frame-level ground truth is unavailable.Source: A. Contributions · p. 1
  • The paper benchmarks four 7B-parameter VLMs for trade-offs among semantic selectivity, latency, hardware demand, and robustness, and proposes deployment guidelines for cloud SOCs and constrained edge systems.Source: A. Contributions · p. 1

B. Roadmap

The paper situates VESSI among video-language and surveillance systems, then compares their capabilities with its operational focus on flexible, traceable forensic surveillance. It distinguishes its reference-free utility score from image-captioning evaluation and anomaly-scoring approaches.

B. Roadmap

  • The roadmap moves from related work and tested models through VESSI, experimental protocol, reference-free measures, results, limitations, and conclusion.Source: B. Roadmap · p. 2

A. Related works

  • UCF-Crime supplies video-level anomaly labels but no natural-language annotations or interactive querying, while UCA adds temporally grounded language tasks with dense spatiotemporal modeling.Source: A. Related works · p. 2
  • Recent anomaly-detection methods focus on anomaly scoring and temporal localization over clips, whereas VESSI emphasizes category-specific frame interrogation, prompt-complexity analysis, model profiling, and traceable records.Source: A. Related works · p. 2
  • Unlike static-image VALOR-EVAL, CMUS targets surveillance videos without fine-grained human ground truth and accounts for detection rate and activation patterns over time.Its hallucination score penalizes persistent over-activation rather than semantic mismatches in captions.Source: A. Related works · p. 3

B. Comparison with the State of the art

III. TESTED VISION LANGUAGE MODELS

The paper benchmarks four similarly sized VLMs and presents VESSI as a modular, frame-based surveillance pipeline. Its evaluation uses video-level labels without exhaustive frame annotations, so it assesses triage utility rather than objective frame-level accuracy.

III. TESTED VISION LANGUAGE MODELS

  • Four approximately 7B-parameter VLMs are benchmarked, with frame-level image models chosen for computational feasibility, flexible integration, and fine-grained prompt control.The authors note that frame-based analysis may not fully capture temporal continuity.Source: §III · pp. 3–4

IV. THE VESSI FRAMEWORK

  • VESSI is a modular, extensible pipeline organized into Acquisition, Analysis, and Threat Evaluation phases.Source: §IV · p. 4

A. Scope and Assumptions

  • Because UCF-Crime provides video-level labels but no exhaustive frame ground truth, VESSI scores describe operational utility and relative model behavior, not temporal localization or objective frame accuracy.Source: A. Scope and Assumptions · p. 4

B. Phase 1 - Acquisition

  • The acquisition phase uses a categorized video dataset, with UCF-Crime selected for evaluation and its outputs assessed qualitatively against category labels and selected-frame content.Source: B. Phase 1 - Acquisition · p. 4

C. Phase 2 - Analysis

VESSI samples video frames and tests category-specific binary prompts at increasing semantic complexity. Affirmative, confidence-filtered frames become traceable alerts for human review, while interpretation and investigative decisions remain with the operator.

C. Phase 2 - Analysis

  • Frames are uniformly sampled at 1 Hz and evaluated with easy, medium, and hard prompts that preserve the same binary detection task while varying semantic complexity.Source: C. Phase 2 - Analysis · p. 5
  • Responses are parsed as Yes, No, or Unclear; Unclear outputs do not trigger alerts, and each response is logged with confidence and system measurements.Source: C. Phase 2 - Analysis · p. 5

D. Phase 3 - Threat Evaluation

  • Only affirmative frames meeting a model-specific confidence threshold are retained, with the experimental threshold set to the tenth-highest confidence among positive responses.Source: D. Phase 3 - Threat Evaluation · p. 5
  • Flagged frames remain linked to their source video and structured metadata, creating traceable alerts for officer review without automating investigative decisions.Source: D. Phase 3 - Threat Evaluation · p. 5

V. EXPERIMENTAL PROTOCOL

The protocol evaluates VLMs in inference mode and normalizes their responses for alerting and metric calculation. Confidence is computed from token probabilities and used to prioritize flagged frames for human review.

V. EXPERIMENTAL PROTOCOL

  • Models run without fine-tuning, and Yes, No, or Unclear responses are normalized; only Yes responses with confidence at or above the threshold enter the alert shortlist.Source: §V · p. 6
  • Per-frame metadata is recorded in structured CSV and plain-text logs for later analysis.Source: §V · p. 6

A. Confidence-based Filtering

VI. COMPOSITE MODEL UTILITY SCORE (CMUS)

CMUS combines video coverage, time savings, detection rate, and a refined hallucination index to compare models when exhaustive frame-level ground truth is unavailable. Its hallucination measure treats persistent affirmative activation as an operational proxy, not proof of incorrect responses.

VI. COMPOSITE MODEL UTILITY SCORE (CMUS)

  • CMUS combines detection coverage, time efficiency, detection rate, and a refined hallucination index into one interpretable model-comparison score.Source: §VI · p. 6, A. Scoring Terms · p. 6

1) Detection Coverage:

2) Temporal Efficiency Analysis:

  • Time savings compare average model inference duration with the videos’ actual playback duration, averaging inference time across three prompt-complexity levels.Source: 2) Temporal Efficiency Analysis · p. 6

3) Detection Rate and Refined Hallucination Index:

B. Final Score Definition

Model-level results show different trade-offs between detection coverage and affirmative activation. The score weighting rewards coverage and time savings while penalizing detection rate and hallucination index.

B. Final Score Definition

  • Coverage and time savings each receive weight 1.0, while detection rate receives −0.5 and the refined hallucination index −1.0.The weighting scheme prioritizes broadly effective and efficient models while discouraging false or misleading detections.Source: B. Final Score Definition · p. 7

VII. RESULTS AND DISCUSSION

  • Figure 2 illustrates successful frame-level detections of a man holding a cashier at gunpoint, a station brawl, and a line of fire.The examples use different prompt difficulties and distinct models.Source: §VII · p. 7

A. Model-level Detection Performance

  • Coverage measures the share of videos with at least one positive response, whereas detection rate measures the average share of analyzed frames marked positive.The reported averages span categories and prompt difficulties.Source: A. Model-level Detection Performance · p. 7
  • InstructBLIP records 99.08% mean detection coverage and a 0.83 per-frame detection rate, while Qwen2.5-VL records 66.9% coverage and a 0.16 detection rate.The results caution that high detection counts may also accompany elevated false-positive rates.Source: A. Model-level Detection Performance · p. 7

B. Hallucination occurrence

Qwen2.5-VL ranks highest by CMUS, while the hallucination analysis distinguishes its selective activation from models with more persistent affirmative responses. All models process video faster than raw playback, with differing time savings.

B. Hallucination occurrence

  • Persistent positive responses across low-activity or ambiguous frames may indicate over-activation; Qwen2.5-VL is described as the most judicious and preferable for high-stakes forensic applications.The paper cautions that isolated affirmative detections should be interpreted in the model's broader activation pattern.Source: B. Hallucination occurrence · pp. 7–8
  • Figure 2 shows successful detections across different actions and prompt difficulties; its confidence values are internal generation scores, not calibrated correctness probabilities.Source: Figure 2 · p. 8
  • Among video–prompt pairs, Qwen2.5-VL had 1,591 below the 10% detection-rate threshold and 12 at ≥95% (H_m = −0.1598), while InstructBLIP had 2,032 at ≥95% (H_m = 1.2030).Source: B. Hallucination occurrence · p. 7
  • DeepSeek-VL has 650 pairs in the 70–95% band and 880 at ≥95% (H_m = 0.7063), while LLaVA has 1,223 below 10% and 123 at ≥95% (H_m = 0.0609).The results describe LLaVA as more selective than DeepSeek-VL.Source: B. Hallucination occurrence · p. 7, Table IV · p. 8

C. Inspection time speed-up

  • 90.96% and 85.32%: InstructBLIP and Qwen2.5-VL save these shares of inspection time, while DeepSeek-VL and LLaVA save 74.13% and 64.59%.All models complete inference faster than raw playback; the evaluated videos average 155.77 minutes.Source: Table V · p. 8, C. Inspection time speed-up · p. 8

D. CMUS Scores

  • 1.6031: Qwen2.5-VL ranks first in CMUS, ahead of LLaVA (1.1909), DeepSeek-VL (0.6826), and InstructBLIP (0.2822).The ranking is consistent with the manually referenced evaluation and the normal-video control.Source: Table VI · p. 8, D. CMUS Scores · pp. 8–9

E. Human-Reference Validation

Human-reference and normal-video evaluations reproduce the same model ordering as reference-free CMUS, while exposing a selectivity–sensitivity trade-off. Computational profiles show that all models process frames faster than raw playback, with different speed and power profiles.

E. Human-Reference Validation

  • The human-reference subset covers 65 videos and 5,909 frames, with 2,342 positive and 3,567 negative frames.Frames were assessed for visually recognizable activity related to each video's category.Source: E. Human-Reference Validation · p. 9
  • The human-reference FPR ordering matches the crime- and normal-video CMUS rankings: Qwen2.5-VL, LLaVA, DeepSeek-VL, then InstructBLIP.The authors describe this agreement as supporting comparative CMUS use within the evaluated setting.Source: E. Human-Reference Validation · p. 9
  • Qwen2.5-VL combines an FPR of 0.0644 with a TPR of 0.2535, while InstructBLIP pairs a TPR of 0.9471 with an FPR of 0.8195.The results expose a sensitivity trade-off between selective and frequent affirmative responses.Source: E. Human-Reference Validation · p. 9
  • LLaVA returned “uncertain” for 4,720 frames (79.88%) under the medium prompt, yielding an FPR of 0.1365 and a TPR of 0.1699.These responses were normalized as Unclear and did not trigger alerts; the passage links this behavior to prompt formulation.Source: E. Human-Reference Validation · p. 9

F. Control Group: Normal Videos

  • The normal-video control reproduces the crime-video CMUS ordering and the human-reference FPR ordering.It treats affirmative responses as false alerts because the 50 videos contain none of the 13 target crime categories.Source: F. Control Group: Normal Videos · p. 9

G. Computational performance

  • InstructBLIP is fastest at approximately 90 ms/frame and draws approximately 170 W; Qwen2.5-VL follows at approximately 145 ms/frame and 215 W.DeepSeek-VL and LLaVA require approximately 260 and 350 ms/frame and draw about 295 and 310 W, respectively; all four use approximately 16.5–17.5 GB VRAM.Source: G. Computational performance · p. 10, Figure 3 · p. 10

H. Answers to the Research Questions

The paper reports that VLMs can retrieve query-related frames and reduce footage for inspection, but model usefulness depends on selectivity and video ambiguity. It presents VESSI as a practical triage and benchmarking framework, while bounding its conclusions by the data, frame-independent pipeline, and evaluated hardware.

H. Answers to the Research Questions

  • VLMs can retrieve frames related to an operator-defined query and substantially reduce footage requiring direct inspection, but usefulness depends on model selectivity and source-video ambiguity.No architecture dominates every operational dimension; the comparison identifies trade-offs in retrieval breadth, selectivity, inference time, and hardware demand.Source: H. Answers to the Research Questions · p. 10
  • Across 5,909 manually referenced frames and the 50-video normal control, reference-free, supervised, and control evaluations produce the same model ordering as CMUS.The authors support comparative use of the score within the evaluated benchmark.Source: H. Answers to the Research Questions · p. 10

VIII. LIMITATIONS AND FUTURE WORK

  • The evaluation inherits UCF-Crime's video-level labels, and its 65-video human-reference subset and 50-video normal control do not provide exhaustive objective frame-level ground truth.Labels may concern ambiguous, overlapping, off-camera, or obscured activity.Source: §VIII · p. 10
  • Frame-independent analysis does not model temporal continuity or guarantee continuous event localization, and the reported hardware profile is specific to one workstation configuration.The experiments use off-the-shelf models without domain-specific fine-tuning.Source: §VIII · p. 10
  • VESSI supports offline post-incident triage and evidence-repository review, as well as configurable compact-model use in edge or UAV deployments.Flagged material remains subject to human assessment.Source: §VIII · p. 10
  • Future work may investigate temporal fusion, adaptive frame sampling, domain-specific adaptation, model compression, and controlled deployment trials.The proposed work would preserve the framework's reference-free and operationally simple nature.Source: §VIII · p. 10

IX. CONCLUSION

  • VESSI retains frames, timestamps, normalized responses, and computational logs for human inspection, with alerts guiding subsequent assessment.The authors position it as a practical triage and benchmarking framework.Source: §IX · pp. 10–11
  • Qwen2.5-VL achieved the highest CMUS (1.6031), followed by LLaVA (1.1909), while broader coverage from DeepSeek-VL and InstructBLIP came with more affirmative responses and lower scores.All models processed frames faster than the configured 1 Hz sampling interval, with average time savings from 64.59% to 90.96%.Source: §IX · pp. 10–11

APPENDIX A SUPPLEMENTARY TECHNICAL AND REPRODUCIBILITY

The appendix records the workstation and software setup, explains per-inference GPU logging, and describes a lightweight Flesch–Kincaid calculation used to quantify prompt readability.

A. Hardware Platform

  • The experiments ran on an Ubuntu 20.04 LTS workstation with a 24-core Intel Core i9-14900KF, 62 GB DRAM, and an NVIDIA RTX 4090 GPU with 24 GB VRAM.The listed setup includes NVIDIA driver 535.183.01 and CUDA support.Source: A. Hardware Platform · p. 13

B. Software Environments

  • Two conda environments separate DeepSeek-VL from InstructBLIP, LLaVA, and Qwen2.5-VL to accommodate differing library configurations and prevent package conflicts.Both use PyTorch GPU builds compiled for CUDA 12.1; model loading uses Hugging Face Transformers interfaces.Source: B. Software Environments · p. 13

C. Custom NVIDIA Management Library (NVML) Logging Module

D. Prompt Readability Assessment

  • The appendix estimates prompt complexity with a lightweight Flesch–Kincaid grade calculation based on sentence length and syllable density.The reported scores are 1.47 for easy, 6.28 for medium, and 11.09 for hard prompts.Source: D. Prompt Readability Assessment · p. 14

E. Detailed Computational Profiles

Supplementary material provides detailed computational profiles and category-level processing times, alongside notation for the paper's evaluation measures and scoring components.

E. Detailed Computational Profiles

F. Category-level Inspection Time

APPENDIX B NOTATION AND SYMBOLS

  • The notation defines coverage, per-frame detection rate, refined hallucination index, time saved, CMUS, and the structured evaluation log.CMUS combines coverage, detection rate, time saved, and hallucination index; the appendix also lists bucket weights and CMUS weights.Source: Appendix B · p. 16
Loading 2610.11674v1…