PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Binary Gating with Server-Side Vision-Language Inference for Surveillance Anomaly Classification

Côme-Alexis Puech, Sébastien Thuau, Amira Gran, Arthur Mennessier, Siba Haidar, Rachid Chelouah

arXiv:2609.07403v1cs.CVcs.AIcs.LG

TL;DR

The paper asks whether federated binary screening with server-side zero-shot VLM inference can approach centralized multiclass utility while reducing raw-video transmission under non-IID data. A lightweight binary classifier is trained federatively to screen videos locally, and flagged videos are sent to a server-side VLM for anomaly-metaclass classification. Fixed-threshold routing achieved F1-macro close to its centralized hybrid counterpart while forwarding 51.4% of test videos; sensitivity-oriented routing increased macro AUC but lowered test-set F1-macro.

  • Problem

    The paper asks whether federated binary screening with server-side zero-shot VLM inference can approach centralized multiclass utility while reducing raw-video transmission under non-IID data.

    Source: §I · p. 1

  • Method

    A lightweight binary classifier is trained federatively to screen videos locally, and flagged videos are sent to a server-side VLM for anomaly-metaclass classification.

    Source: A. Problem Setting and Overall Design · p. 2

  • Results

    Fixed-threshold routing achieved F1-macro close to its centralized hybrid counterpart while forwarding 51.4% of test videos; sensitivity-oriented routing increased macro AUC but lowered test-set F1-macro.

    Source: §V · p. 6, §VI · p. 7

  • Takeaways & Limitations

    Routing acts as a controllable mechanism to balance server-side VLM usage and anomaly sensitivity.

    Source: §VI · p. 7

    The study is limited to a single Dirichlet partition, heuristic routing rules, and FedAvg-based optimization in a three-node heterogeneous deployment.

    Source: §VI · p. 7, §V · p. 7

Abstract

from arXiv · show

Privacy-sensitive surveillance systems could benefit from large vision-language models (VLMs), but such models typically require centralized access to raw video. In federated learning settings, this challenge is amplified by non-independent and identically distributed (non-IID) client data, which can make direct multiclass anomaly classification unstable, especially for rare categories. We propose a hybrid two-stage architecture that combines a federated binary convolutional neural network (CNN) gate with server-side zero-shot VLM inference. The lightweight LiteCNN3D gate performs local anomaly screening and forwards only flagged videos to Qwen3-VL-8B, which assigns them to four anomaly metaclasses. We evaluate this design on UCF-Crime grouped into five coarse metaclasses and implement the federated stage in a real three-node heterogeneous deployment. In the studied setting, direct federated multiclass training collapses, whereas the proposed decomposition yields a better trade-off between classification quality and raw-video transmission. With fixed-threshold routing, the federated hybrid pipeline preserves nearly the same macro-averaged F1 score (F1-macro) as its centralized CNN+VLM counterpart while reducing the fraction of transmitted videos to 51.4%, although with a lower proxy macro receiver operating characteristic area under the curve (ROC AUC) than the centralized hybrid system. A complementary sensitivity-oriented routing operating point increases macro ROC AUC from 0.673 to 0.692 and reduces the false negative rate from 29.3% to 22.9%, but decreases F1-macro from 0.503 to 0.485 while increasing transmission from 51.4% to 57.9%. These results suggest that federation is better suited to coarse local screening, while routing rules can be adjusted to trade server-side VLM usage for higher anomaly sensitivity.

I. INTRODUCTION

The paper asks whether federated binary screening plus server-side zero-shot VLM inference can preserve classification utility while reducing raw-video transmission under non-IID data. In the studied setting, direct federated multiclass training collapses, while the hybrid approach remains usable and is evaluated in a real heterogeneous deployment.

I. INTRODUCTION

  • The federated hybrid preserves nearly the centralized counterpart’s F1-macro while substantially reducing raw-video transmission in the studied setting.The comparison is on UCF-Crime grouped into five coarse metaclasses.Source: §I · p. 1
  • The proposed two-stage design uses federated binary edge screening and server-side zero-shot VLM classification for flagged videos.This decomposition is presented as an alternative to direct federated multiclass classification under non-IID data.Source: §I · p. 1
  • The federated stage is evaluated in a real three-node heterogeneous deployment that includes a CPU-only client.Source: §I · p. 1
  • The study compares fixed-threshold and sensitivity-oriented routing to examine the classification–transmission trade-off under non-IID data.Source: §I · p. 1

II. RELATED WORK

Prior surveillance anomaly work largely evaluates centralized anomaly scoring or localization, while federated learning faces instability under heterogeneous client data. The paper positions its contribution as a federated gate that filters videos before server-side VLM inference, evaluated on a physical multi-node deployment.

A. Anomaly Detection in Surveillance Video

  • Centralized UCF-Crime methods mainly target anomaly scoring or localization, and assume centralized access to footage.Examples include UCF-Crime, RTFM, MGFN, UR-DMU, CLIP-TSA, and VadCLIP.Source: A. Anomaly Detection in Surveillance… · p. 2

B. Federated Learning for Surveillance Video Analysis

  • Federated surveillance approaches include caption-based semantic distillation and prompt-based multimodal learning, whereas this work keeps the VLM outside federated training.The system is evaluated in a physical heterogeneous multi-node deployment rather than through server-side client emulation.Source: B. Federated Learning for Surveillance… · p. 2

C. Vision-Language Models for Surveillance Anomaly Recognition

  • Prior VLM approaches assume centralized inference over full visual inputs; the paper identifies a federated gate followed by server-side VLM inference as an unexamined design.Examples include LAVAD, VERA, and Holmes-VAU.Source: C. Vision-Language Models for… · p. 2

III. METHODOLOGY

The method targets privacy-sensitive surveillance with heterogeneous clients, keeping lightweight screening on-device and assigning flagged videos to a server-side VLM. Experiments use grouped UCF-Crime data with five metaclasses and video-level splits.

A. Problem Setting and Overall Design

  • A federated binary gate screens videos locally; flagged videos go to a server-side VLM for assignment to one of four anomaly metaclasses, while normal predictions remain on-device.Source: A. Problem Setting and Overall Design · p. 2

B. Dataset and Metaclass Grouping

  • UCF-Crime categories are grouped into five metaclasses, with all anomalous categories merged into one Anomaly class for binary routing.The grouped categories are Destruction, PropertyCrime, Violence, RoadAccidents, and NormalVideos.Source: B. Dataset and Metaclass Grouping · p. 2
  • NormalVideos accounts for 49.7% of training videos, while Destruction accounts for 4.3%, leaving minority metaclasses vulnerable to low client-level support.The reported video-level distribution has 1,610 training and 290 test videos; splits and validation selection are performed at video level.Source: B. Dataset and Metaclass Grouping · p. 2, Table I · p. 3

C. LiteCNN3D Backbone

CNN systems share a LiteCNN3D backbone with task-specific binary or multiclass heads. Federated experiments use a single heterogeneous partition, and the hybrid pipeline sends locally flagged videos to the server for metaclass classification.

C. LiteCNN3D Backbone

  • LiteCNN3D is the shared CNN backbone, with task-specific output heads of C = 2 for binary routing and C = 5 for multiclass classification.The network has approximately 3M parameters and uses five 3D convolutional blocks; video predictions average clip-level logits over extracted windows.Source: C. LiteCNN3D Backbone · p. 3, Figure 1 · p. 3

D. Federated Data Partitioning

  • Federated experiments use Dirichlet-based non-IID partitioning with α = 0.30 across three clients, with the same α in binary and multiclass settings.Because results use one Dirichlet draw, they are a case study of one heterogeneous partition rather than an average across partitions.Source: D. Federated Data Partitioning · p. 3

E. Two-Stage Hybrid Pipeline

  • The federated binary LiteCNN3D gate keeps predicted-normal videos on-device and sends flagged videos to Qwen3-VL-8B for classification into four anomaly metaclasses.Source: E. Two-Stage Hybrid Pipeline · p. 3

1) Stage 1 — Federated Binary CNN Gate:

The federated binary gate screens videos locally, forwarding flagged clips for server-side zero-shot classification into anomaly metaclasses. The gate’s threshold favors reduced communication, while the server stage performs semantic classification on the routed subset.

1) Stage 1 — Federated Binary CNN Gate:

  • The federated binary LiteCNN3D gate screens videos locally using a sigmoid score thresholded at τ = 0.7.The fixed threshold is a conservative communication-oriented operating point.Source: 1) Stage 1 — Federated Binary CNN Gate · p. 3
  • Raising the gate threshold reduces unnecessary server forwarding and prioritizes communication and inference cost over maximum anomaly recall.The gate is a transmission filter rather than the final anomaly classifier.Source: 1) Stage 1 — Federated Binary CNN Gate · p. 3

2) Stage 2 — Server-Side Vision-Language Model:

  • Videos flagged by the gate are sent to Qwen3-VL-8B, which uses 24 ordered, uniformly sampled frames to assign one of four anomaly metaclasses.Videos rejected by the gate are labeled NormalVideos and are not transmitted.Source: 2) Stage 2 — Server-Side… · pp. 3–4
  • The two-stage pipeline sends only gate-flagged videos across the privacy boundary for server-side zero-shot VLM classification.Figure 2 depicts the local gate followed by server-side classification.Source: Figure 2 · p. 4, 2) Stage 2 — Server-Side… · p. 4

IV. EXPERIMENTS AND RESULTS

The experiments compare centralized and federated CNNs, a standalone VLM, and centralized and federated hybrid pipelines to isolate federation, task granularity, and server-side semantic reasoning.

A. Compared Systems

  • Seven systems span centralized and federated binary and multiclass CNNs, a standalone zero-shot VLM, and centralized and federated CNN+VLM hybrids.The centralized hybrid is the nonfederated counterpart to the proposed federated hybrid.Source: A. Compared Systems · p. 4
  • The binary LiteCNN3D models provide centralized and federated references for anomaly-versus-normal screening, with the federated model also assessed as a routing stage.The independent federated-gate evaluation tests whether binary screening remains viable under non-IID client data.Source: A. Compared Systems · p. 4
  • The centralized and end-to-end federated multiclass LiteCNN3D models compare direct five-class prediction under pooled and non-IID federated training.The federated multiclass model is the direct federated alternative to task decomposition.Source: A. Compared Systems · p. 4
  • The standalone server-side Qwen3-VL-8B classifies all 290 test videos, while each hybrid pairs a binary gate with the same server-side VLM.The standalone VLM is evaluated without transmission filtering; the proposed federated hybrid forwards only locally flagged videos.Source: A. Compared Systems · p. 4

B. Evaluation Protocol

The evaluation reports video-level gate and multiclass metrics, selects sensitivity-oriented routing on validation data, and treats hybrid and standalone VLM AUC values as proxy measures.

B. Evaluation Protocol

  • Binary-gate metrics are computed at video level after averaging clip logits, and include accuracy, F1-binary, ROC AUC, ECE, FPR, and FNR.ECE is lower-is-better.Source: B. Evaluation Protocol · p. 4
  • Multiclass and hybrid results use video-level metrics including macro ROC AUC, per-class AUC and recall, F1-macro, and fraction transmitted.Unless stated otherwise, results use the final available checkpoint without test-set checkpoint selection.Source: B. Evaluation Protocol · p. 4
  • Temperature scaling and sensitivity-oriented routing thresholds are selected using validation data from the original training split, not the test set.The official test set is reserved for final evaluation.Source: B. Evaluation Protocol · pp. 4–5
  • Hybrid macro ROC AUC uses proxy five-class score vectors derived from the binary gate score and final label, not calibrated class probabilities.Standalone VLM AUC uses one-hot hard predictions and is only approximately comparable with CNN softmax or hybrid proxy AUC.Source: B. Evaluation Protocol · p. 5

C. Federated Training Setup

In the real heterogeneous deployment, the federated hybrid preserves nearly the centralized hybrid’s F1-macro while transmitting fewer videos, whereas direct federated multiclass learning collapses in this setting.

C. Federated Training Setup

  • Federated training uses Flower in a real three-node heterogeneous deployment, including two GPU clients and one CPU-only client.FedAvg runs for 20 communication rounds with 2 local epochs per round; synchronous rounds wait for the slowest client.Source: C. Federated Training Setup · p. 5

D. Binary Routing Performance

  • The centralized gate reaches 0.625 accuracy and 0.693 F1-binary, while the federated gate reaches 0.709 accuracy and 0.729 F1-binary.The centralized gate retains higher ROC AUC (0.790 vs. 0.766) and better calibration (ECE 0.103 vs. 0.224).Source: Table II · p. 5, D. Binary Routing Performance · p. 5

E. Multiclass and Hybrid System Comparison

  • F-CNN+VLM preserves nearly the centralized hybrid’s F1-macro (0.503 vs. 0.506) while reducing transmitted videos from 79.7% to 51.4%.Table III compares global video-level multiclass systems; the standalone VLM has the highest macro ROC AUC, with values based on one-hot hard predictions interpreted cautiously.Source: Table III · p. 5, E. Multiclass and Hybrid System… · p. 5
  • F-CNN-Multi reaches only 0.048 F1-macro and has the weakest overall performance, unlike the usable federated binary screening in this studied non-IID setting.The paper’s main result is that federated learning remains usable for coarse screening but not direct end-to-end multiclass classification.Source: E. Multiclass and Hybrid System… · p. 5
  • C-CNN+VLM has the strongest recall on most anomaly classes, while F-CNN+VLM retains recall of 0.667 for NormalVideos, 0.633 for PropertyCrime, and 0.652 for RoadAccidents.Standalone VLM has the highest AUC on most classes; F-CNN-Multi recall is 0.000 on NormalVideos, PropertyCrime, and RoadAccidents.Source: E. Multiclass and Hybrid System… · pp. 5–6, Table IV · p. 6

F. Sensitivity-Oriented Routing for Hybrid Inference

The sensitivity-oriented rule combines calibrated anomaly scores with predictive entropy to forward uncertain or borderline videos for server-side VLM inference. It improves anomaly sensitivity while increasing transmission and lowering F1-macro, making routing a trade-off rather than a uniformly better replacement for fixed-threshold routing.

F. Sensitivity-Oriented Routing for Hybrid Inference

  • The rule combines temperature-calibrated anomaly scores with predictive entropy, forwarding a video unless both fall below selected thresholds.It is an exploratory sensitivity-oriented operating point, not an ablation isolating entropy's contribution.Source: F. Sensitivity-Oriented Routing for… · p. 6
  • The selected configuration uses T = 1.2, plow = 0.5, and hlow = 0.69, with temperature scaling and threshold selection performed on validation data from the original training split.Because the entropy threshold is near maximum binary entropy, the point is primarily a more permissive routing configuration rather than evidence that entropy alone improves routing.Source: F. Sensitivity-Oriented Routing for… · p. 6
  • Recall rises for RoadAccidents (65.2% to 73.9%), Destruction (16.7% to 26.7%), and Violence (26.3% to 31.6%), while NormalVideos recall falls from 66.7% to 60.0%.The lower NormalVideos recall is consistent with the higher transmission rate.Source: F. Sensitivity-Oriented Routing for… · p. 6
  • Macro AUC rises from 0.673 to 0.692 and false negative rate falls from 29.3% to 22.9%, while test F1-macro falls from 0.503 to 0.485 and transmission rises from 51.4% to 57.9%.Table V compares fixed-threshold and sensitivity-oriented routing; SENT (%) is the fraction of test videos forwarded to the server.Source: Table V · p. 6, F. Sensitivity-Oriented Routing for… · p. 6
  • Fixed-threshold routing favors F1-macro and transmission cost, whereas sensitivity-oriented routing forwards more videos and reduces missed anomalies.The results present controllable operating points, not universal deployment rates.Source: F. Sensitivity-Oriented Routing for… · p. 6

V. DISCUSSION

In the studied non-IID setting, the federated binary gate remained operational while direct federated multiclass training collapsed. The discussion frames the hybrid system as a deployment-oriented trade-off whose conclusions are bounded by the data, routing, optimization, and metric choices.

V. DISCUSSION

  • Under the same non-IID partitioning and training conditions, the end-to-end federated multiclass CNN collapsed, whereas the federated binary gate remained operational.The paper interprets this as evidence that task decomposition is more suitable than direct federated multiclass learning in the studied setting.Source: §V · p. 6
  • Binary screening requires a coarser decision boundary, while multiclass classification must separate imbalanced categories that may be unevenly represented or absent across clients.The paper states that client drift and minority-class underrepresentation affect direct federated multiclass learning more strongly.Source: §V · p. 6
  • Lightweight CNN inference can run on heterogeneous edge clients, including CPU-only devices, while the centralized VLM reasons semantically only over videos selected by the local gate.The VLM is kept outside the federated loop, reducing raw-video transmission compared with sending every video for server-side inference.Source: §V · p. 6
  • Fixed-threshold routing forwards 51.4% of test videos, while sensitivity-oriented routing forwards 57.9%, improving macro AUC and false negative rate but lowering test F1-macro.The sensitivity-oriented point prioritizes anomaly sensitivity rather than providing a uniformly better routing strategy.Source: §V · p. 6
  • Because UCF-Crime contains more anomalous videos than typical surveillance streams, both measured transmission rates may overestimate those in real deployments.Source: §V · p. 7
  • Centralized baselines are upper references rather than target operating points; the stated objective is useful operation under privacy, communication, and edge-resource constraints despite a residual performance gap.Source: §V · p. 7
  • The study uses a single Dirichlet partition, heuristic routing, and FedAvg under short local training, without characterizing variability across partitions, routing configurations, or federated optimizers.The sensitivity-oriented rule is one operating point without a dedicated ablation of its entropy component, and its thresholds use validation data from the original training split.Source: §V · p. 7
  • Macro AUC is not strictly comparable across systems because CNNs provide continuous scores, whereas VLM and hybrid systems use hard labels or proxy score vectors.The paper frames its contribution as deployment-oriented rather than a state-of-the-art classification claim.Source: §V · p. 7

VI. CONCLUSION

The federated hybrid pipeline approached its centralized counterpart's F1-macro while reducing transmission under fixed-threshold routing, and sensitivity-oriented routing exposed a different sensitivity–transmission trade-off. The conclusion presents federation as better suited to coarse local screening than direct fine-grained semantic classification, within a limited experimental scope.

VI. CONCLUSION

  • The federated hybrid pipeline achieved F1-macro close to its centralized hybrid counterpart while forwarding only 51.4% of test videos under fixed-threshold routing.Source: §VI · p. 7
  • Sensitivity-oriented routing increases macro ROC AUC and reduces false negative rate, but lowers F1-macro and increases raw-video transmission.The paper describes routing as a controllable mechanism for balancing VLM usage and anomaly sensitivity.Source: §VI · p. 7
  • The results support federation for coarse local anomaly screening rather than direct fine-grained semantic classification.The design filters raw videos locally with a lightweight CNN and transmits selected videos for centralized VLM-based semantic classification.Source: §VI · p. 7
  • The study is limited to a single Dirichlet partition, heuristic routing rules, and FedAvg-based optimization in a three-node heterogeneous deployment.Source: §VI · p. 7

Related papers

Loading 2609.07403v1…