PaperlayerView exact revision on arXiv

Source-linked AI summary

Parser-Free VLM Verification for Federated Weakly Supervised Video Anomaly Detection

Sébastien Thuau, Amira Gran, Siba Haidar, Rachid Chelouah

arXiv:2609.07455v1cs.CVcs.AIcs.LG

TL;DR

Weakly supervised VAD must localize anomalies from video-level labels while surveillance data remain distributed, and the paper asks how frozen-VLM feedback can be efficient and parser-free. It proposes a federated MIL scorer with selective frozen-VLM verification and compares parsed text generation with next-token logits. The logit interface improves frame-level AUC and AP over the MIL baseline across both VLMs without temporal smoothing in its main configuration.

  • Problem

    Weakly supervised VAD relies on coarse video-level supervision, while the paper asks how frozen-VLM feedback can remain efficient, parser-free, and compatible with causal segment-level scoring.

    Source: §I · p. 1

  • Method

    The paper trains a compact federated MIL head and uses a frozen VLM to verify selected suspect segments, including through next-token Yes/No logits.

    Source: §I · p. 1, 2) Logit-Based Interface · p. 3

  • Results

    The parser-free logit interface improves both F-AUC and F-AP over the MIL baseline across both VLMs without temporal smoothing in its main configuration.

    Source: §V · p. 6

  • Takeaways & Limitations

    With λ = 1, next-token logit feedback is causal at the segment-scoring level and offers a parser-free alternative to text-generation verification.

    Source: §I · p. 1, §V · p. 6

    Evaluation is limited to centralized offline VLM evaluation, UCF-Crime, and four physical CPU-only clients.

    Source: §V · p. 6

Abstract

from arXiv · show

How can vision-language models help video anomaly detection (VAD) when surveillance data remain distributed, weakly labeled, and resource-constrained? Most weakly supervised VAD methods assume centralized training; recent VLM-based extensions further rely on dense inference, generated explanations, or additional adaptation. We introduce a lightweight federated MIL-VLM cascade in which only a compact MIL scorer is trained across clients, while a frozen VLM verifies high-scoring suspect segments post hoc. We study two VLM feedback interfaces: parsed text-generation decisions and a logit-based interface that extracts a continuous anomaly score from next-token Yes/No probabilities. Experiments on UCF-Crime with InternVL3.5-2B and Qwen3-VL-2B-Instruct show that text-generation verification can improve frame-level AUC after diagnostic temporal post-processing, but remains sensitive to prompts, parsers, model choice, and smoothing. In contrast, the logit interface provides a fixed parser-free signal that improves both frame-level AUC and frame-level AP over the MIL baseline across both VLMs, without temporal post-processing in its main configuration. Since suspect segments are updated independently once available, next-token logit feedback provides a simple segment-local alternative to text-generation verification.

I. INTRODUCTION

Weakly supervised VAD must localize events from video-level labels even when surveillance data are distributed and difficult to centralize. The paper proposes a lightweight federated MIL scorer with selective frozen-VLM verification, comparing text-generation and parser-free logit feedback.

I. INTRODUCTION

  • Distributed surveillance data, video-level-only labels, and the need for frame- or segment-level localization motivate compact federated weakly supervised VAD.Raw footage can be difficult to centralize because of privacy and bandwidth constraints, while client devices may have limited computational power.Source: §I · p. 1
  • The paper asks how frozen-VLM feedback can remain efficient, parser-free, and compatible with causal segment-level scoring.Source: §I · p. 1
  • A lightweight federated MIL–VLM cascade trains only a compact MIL head across clients and uses a frozen VLM to verify selected suspect segments post hoc.The VLM is not trained, synchronized, or aggregated across clients.Source: §I · p. 1
  • Text-generation VLM verification is sensitive to prompting, parsing, model choice, and temporal post-processing.Source: §I · p. 1
  • The logit-based interface extracts a continuous anomaly score from next-token Yes/No probabilities, avoids keyword parsing, and preserves ranking information.Its main configuration remains causal at the segment-scoring level.Source: §I · p. 1

II. RELATED WORK

Prior work studies weakly supervised, federated, and VLM-based approaches to video anomaly detection, with different assumptions about training and inference. This paper centers on selective verification by a frozen VLM and on how its feedback is represented.

A. Weakly Supervised Video Anomaly Detection

  • Weakly supervised VAD trains from video-level labels but requires frame- or segment-level localization; prior methods add temporal modeling, feature magnitude learning, memory, or bias reduction.Sultani et al. introduced the MIL formulation for WSVAD on UCF-Crime.Source: A. Weakly Supervised Video Anomaly… · p. 1

B. Federated Video Anomaly Detection

  • Prior federated VAD methods include semantic distillation, multimodal prompt learning, and local-to-global experts, whereas this work federates only a lightweight MIL scorer and keeps the VLM frozen.The paper's main question is the interface between that scorer and the frozen VLM.Source: B. Federated Video Anomaly Detection · p. 2

C. Vision-Language Models for Video Anomaly Detection

  • VLM-based VAD has explored generated descriptions, rule-based reasoning, guiding questions, and chain-of-thought prompts, while Holmes-VAU and SlowFastVAD focus VLM computation on informative regions.SlowFastVAD is closest in spirit but is centralized and uses retrieval-augmented reasoning rather than logit-based verification; this work studies next-token likelihoods as a continuous verification interface for weakly supervised VAD.Source: C. Vision-Language Models for Video… · p. 2
  • The two-stage cascade routes suspect segments from a federated MIL scorer to a frozen VLM, while non-suspect segments retain their MIL scores.Source: Figure 1 · p. 2

III. METHODOLOGY

The method uses a two-stage pipeline for weakly supervised VAD, where video-level training labels support segment scoring and segment scores are expanded for frame-level evaluation. Federated learning trains the MIL scorer, and a frozen VLM selectively rescores suspect segments.

III. METHODOLOGY

B. Federated MIL Anomaly Scorer

A compact MIL head learns segment anomaly scores from frozen CLIP features using video-level supervision and federated training. A frozen VLM selectively verifies suspect segments, with text-generation feedback parsed into confirmation or attenuation decisions.

B. Federated MIL Anomaly Scorer

  • The MIL scorer uses frozen CLIP ViT-B/16 segment features and a compact MLP head, trained federatively with video-level labels and regularization.It averages the top k = 3 segment scores for video-level prediction and is trained with FedAvg.Source: B. Federated MIL Anomaly Scorer · pp. 2–3

C. Selective VLM Verification

  • Segments with MIL scores above a fixed threshold are sent to the VLM; other segments keep their MIL scores, and the paper studies generated-text parsing and next-token Yes/No logits.Source: C. Selective VLM Verification · p. 3

1) Text-Generation Interface:

  • Text-generation responses containing “anomalous” or “abnormal” confirm the MIL score, while responses parsed as normal attenuate it and unparsed responses leave it unchanged.The keyword rule does not handle negations and may misclassify some chain-of-thought outputs; a 0–10 score prompt is also evaluated diagnostically when valid.Source: 1) Text-Generation Interface · p. 3

2) Logit-Based Interface:

The logit interface turns the frozen VLM’s next-token Yes/No logits into a continuous anomaly score, updating suspect segments independently in the main configuration. The accompanying score-processing section contrasts this with text-generation corrections and diagnostic temporal smoothing.

2) Logit-Based Interface:

  • The fixed normality prompt maps the next-token No logit to anomaly and uses a restricted Yes/No softmax as a ranking signal, not a calibrated probability.Candidate token IDs are fixed for each tokenizer and verified as single-token continuations under the model’s inference template.Source: 2) Logit-Based Interface · p. 3
  • The logit interface yields a continuous, parser-free anomaly score and updates each suspect segment independently without temporal smoothing in the main λ = 1 configuration.At λ = 1, the VLM anomaly probability replaces the suspect segment’s MIL score.Source: 2) Logit-Based Interface · p. 3, D. Score Substitution, Fusion, and… · p. 3

D. Score Substitution, Fusion, and Temporal Post-Processing

  • Text-generation verification attenuates segments parsed as normal to 0.1 times their MIL score, while confirmed anomalies and unparsed responses remain unchanged.Non-suspect segments retain their MIL scores.Source: D. Score Substitution, Fusion, and… · p. 3
  • The diagnostic text-generation sweep tests linear-fusion weights, EMA decay values, majority-vote windows, and smoothing over either the full timeline or suspect sequence; the main logit configuration uses no smoothing.Offline smoothing may be bidirectional or centered, while causal smoothing uses forward-only EMA and past-only neighborhoods.Source: D. Score Substitution, Fusion, and… · p. 3

A. Experimental Setup and Dataset

The experiments use UCF-Crime video-level labels for training and frame-level annotations for evaluation, with a federated MIL stage followed by offline verification of selected segments. The setup also quantifies VLM routing and notes that centralized verification is not inherently privacy-preserving.

A. Experimental Setup and Dataset

  • The test set contains 290 UCF-Crime videos with frame-level anomaly annotations, while training uses video-level labels.UCF-Crime contains normal videos and 13 anomaly categories.Source: A. Experimental Setup and Dataset · p. 3
  • Frame-level AUC and average precision are computed by assigning each frame the score of its corresponding one of 32 temporal segments.The table reports test-set F-AUC/F-AP percentages and the number of runs.Source: A. Experimental Setup and Dataset · p. 3, Table I · p. 4
  • The federated Stage 1 scorer trains only a MIL head on frozen CLIP features across four CPU-only clients using 30 FedAvg rounds.Training videos are partitioned by a Dirichlet-based allocation over video-level labels with α = 0.5; each round uses one local epoch.Source: A. Experimental Setup and Dataset · p. 3
  • VLM verification is conducted offline after federated training, so centralized frame transfer is not necessarily privacy-preserving, though clients with sufficient VLM capability can run it locally or on-premise.Two frames resized to 448 × 448 are sampled per suspect segment; the evaluated frozen models are InternVL3.5-2B [19] and Qwen3-VL-2B-Instruct [20].Source: A. Experimental Setup and Dataset · pp. 3–4
  • At τ = 0.5, the verifier receives an average of 8.09 ± 0.41 segments per video, or 25.3% ± 1.3% of segments, reducing VLM calls 3.95× versus dense verification.Across the five Stage 1 checkpoints, 51.7% of test videos have no suspect segment and skip VLM verification.Source: A. Experimental Setup and Dataset · p. 4

B. Stage-1 Backbone Screening and Checkpoint Selection

The backbone screen selects federated CLIP+MLP for the cascade, while the text-generation prompt sweep finds limited raw gains and sensitivity to strategy. The screening is not presented as a state-of-the-art comparison, and the prompt sweep is diagnostic rather than a validation protocol.

B. Stage-1 Backbone Screening and Checkpoint Selection

  • CLIP+MLP exceeds the best 3D CNN by 8.7 F-AUC points centrally and 8.1 points federated, with the same F-AP trend; federated training drops F-AUC by 0.8 points versus centralized training.The authors use CLIP+MLP as Stage 1; they did not expand CNN baselines into a multi-seed benchmark because the gap was already substantial.Source: B. Stage-1 Backbone Screening and… · p. 4
  • Dense zero-shot VLM scoring falls below the federated CLIP+MLP baseline, with Qwen3 basic nearly 28 F-AUC points lower and labeling 88.6% of segments anomalous.InternVL labels 23.3% of segments anomalous; VERA also degrades both VLMs in dense use, despite being Qwen3’s strongest cascade prompt in Table III.Source: B. Stage-1 Backbone Screening and… · p. 4
  • The screening is not a state-of-the-art claim: reported FedVAD and FedWSAD results are not directly comparable because their client splits and added mechanisms differ.FedVAD reports 85.13% weakly supervised AUC, and FedWSAD reports 84.06% federated AUC on its random split.Source: B. Stage-1 Backbone Screening and… · p. 4

C. Text-Generation Strategy Exploration

  • R13 has the largest raw gain but its numeric outputs are discretized, while chain-of-thought responses mix normal and anomalous cues that make keyword parsing unstable.InternVL mostly uses scores 3, 5, and 7 rather than the full 0–10 scale; VERA-style prompts do not transfer uniformly to selective verification.Source: C. Text-Generation Strategy Exploration · pp. 4–5
  • In a diagnostic evaluation of text-generation verification, only 5 of 14 tested strategies improve over the MIL baseline, showing that gains are limited across the evaluated prompts.The experiment used InternVL3.5-2B on one high-performing federated Stage 1 checkpoint, with the raw correction rule and no temporal post-processing.Source: C. Text-Generation Strategy Exploration · pp. 4–5

D. Cross-Model Text-Generation Verification and Post-Processing

Across five Stage 1 checkpoints, text-generation verification is sensitive to prompt and model choices, while temporal post-processing can improve F-AUC but complicates attribution of those gains.

D. Cross-Model Text-Generation Verification and Post-Processing

  • The evaluation spans five retained prompt strategies, multiple VLM backbones, five Stage 1 checkpoints, and settings with or without temporal post-processing.The strategies include binary prompts, contextual and debiased variants, and a score-output prompt.Source: D. Cross-Model Text-Generation… · p. 5
  • +1.553 ± 0.211 pp for InternVL R11+PP and +1.541 ± 0.257 pp for Qwen3 R5+PP are the strongest text-generation results.Table III reports F-AUC changes over five Stage 1 checkpoints; +PP denotes diagnostic F-AUC-selected temporal post-processing.Source: D. Cross-Model Text-Generation… · p. 5, Table III · p. 5
  • Raw text-generation corrections give modest, inconsistent gains, whereas temporal post-processing substantially improves several configurations when optimizing F-AUC.The post-processed results are a diagnostic upper bound, not a deployment-ready model-selection protocol.Source: D. Cross-Model Text-Generation… · p. 5
  • Post-processing rankings conflate VLM signal quality with temporal regularization, and Qwen3 R13 requires model-specific parser adaptation.The reported F-AP is limited to the best configuration per VLM in Table IV.Source: D. Cross-Model Text-Generation… · p. 5, Table IV · p. 5

E. Text-Generation versus Logit-Based Verification

The paper compares each VLM’s strongest post-processed text-generation configuration with fixed logit verification without temporal smoothing. Logit verification produces larger F-AP gains, while the logit interface also offers a parser-free, segment-local signal.

E. Text-Generation versus Logit-Based Verification

  • The comparison uses each VLM’s strongest post-processed text-generation configuration against the main logit configuration, λ = 1, without temporal smoothing.Text-generation results use their most favorable diagnostic setting, whereas logit verification uses a fixed rule.Source: E. Text-Generation versus Logit-Based… · p. 5
  • Logit verification yields F-AP improvements 4.4× larger for InternVL and 5.0× larger for Qwen3 than the best text-generation configuration, with approximately 84.91% and 85.38% F-AUC, respectively.The approximate F-AUC values use the mean Stage 1 baseline of 83.86%.Source: E. Text-Generation versus Logit-Based… · p. 5
  • For λ = 1 logit verification without temporal smoothing, causal and bidirectional scores are identical in the segment-scoring sense.The paper presents next-token logits as a segment-local, parser-free signal that preserves continuous ordering among suspect segments.Source: Table V · p. 6, E. Text-Generation versus Logit-Based… · p. 6

F. Causal versus Bidirectional Evaluation

The logit interface improves F-AUC and F-AP over the MIL baseline across both VLMs without temporal smoothing, and is causal at the segment-scoring level when λ = 1. Text-generation verification loses F-AUC under the causal constraint, while the conclusion notes limits in evaluation scope and setup.

F. Causal versus Bidirectional Evaluation

  • All post-processed text-generation configurations lose F-AUC under the causal constraint, while the logit verifier's bidirectional and causal results are identical by construction.The logit configuration uses λ = 1 and no temporal smoothing.Source: F. Causal versus Bidirectional… · p. 6

V. CONCLUSION

  • Only a compact MIL head is trained federatively on frozen CLIP features; the frozen VLM verifies selected suspect segments post hoc and remains outside federated optimization.Source: §V · p. 6
  • The parser-free logit interface improves both F-AUC and F-AP over the MIL baseline across both VLMs without temporal smoothing, and is causal at the segment-scoring level when λ = 1.Source: §V · p. 6
  • Evaluation is limited to UCF-Crime, four physical CPU-only clients, centralized offline VLM evaluation, pre-extracted CLIP features, and fixed routing and fusion settings.The client data use Dirichlet partitions rather than naturally occurring deployment silos.Source: §V · p. 6
Loading 2609.07455v1…