Source-linked AI summary
Real-Time Video Anomaly Detection Using YOLO Pose Estimation and CLIP-Based Semantic Scoring
Vanodhya G. Warnasooriya, Amir Hajian, Watchara Ruangsang, Supavadee Aramvith
TL;DR
The paper addresses real-time person-level video anomaly detection for CCTV while reducing the complexity of multi-component pipelines. It combines YOLO v11n-pose detection and pose estimation with CLIP text-image similarity scoring, achieving higher throughput while maintaining comparable benchmark AUROC and improving performance on the targeted indoor dataset. The method remains bounded by fixed prompt coverage and indoor-scene detector fine-tuning.
Problem
CCTV monitoring motivates automated detection of falls, altercations, and abnormal postures because human operators experience attention fatigue across multiple feeds.
Method
A two-stage framework uses YOLO v11n-pose for person detection and pose estimation, then CLIP ViT-B/32 for text-prompt semantic anomaly scoring.
Results
51.39 FPS delivers a 3.36× throughput gain over the prior baseline while maintaining comparable AUROC on established benchmarks and reaching 84.13% on the targeted indoor dataset.
Takeaways & Limitations
Direct CLIP text-image similarity can serve as an effective scoring mechanism for lightweight real-time person-level anomaly detection.
Takeaways & Limitations
The fixed four-prompt set limits detectable anomaly categories, while indoor-scene fine-tuning may reduce generalization to markedly different environments.
Abstract
from arXiv · showhide
We propose a lightweight two-stage framework for real-time video anomaly detection. The first stage employs YOLO v11n-pose to detect persons and extract seventeen skeletal keypoints in a single forward pass. The second stage encodes each cropped person region through CLIP ViT-B/32 and computes cosine similarity against predefined textual descriptions of anomalous behaviors. This architecture eliminates the need for optical flow, standalone pose estimators, and density-based scoring modules. Experiments on CUHK Avenue, ShanghaiTech Campus, and a custom indoor dataset collected at Chulalongkorn University demonstrate an end-to-end throughput of approximately 51 FPS on an NVIDIA Titan XP GPU, a 3.36x speedup over the multi-feature baseline, while maintaining frame-level AUROC values of 89.26%, 70.26%, and 84.13%, respectively.
I. INTRODUCTION
The paper presents a lightweight real-time anomaly detector for indoor CCTV that combines YOLO v11n-pose with CLIP semantic scoring. It streamlines prior multi-component approaches while targeting multiple person-level anomaly categories and real-time deployment.
- Human attention fatigue in multi-feed CCTV monitoring motivates automated detection of falls, altercations, and abnormal postures.
- Prior multi-feature pipelines combining AlphaPose, FlowNet2, CLIP, and GMM/kNN scoring reach approximately 15 FPS, limiting throughput.
- YOLO v11n-pose performs detection and pose estimation while CLIP ViT-B/32 provides semantic anomaly scoring in the proposed two-stage framework.
- 3.36× throughput gain over the prior baseline is reported for the streamlined pipeline using direct CLIP semantic scoring instead of multi-feature density estimation.
- Text prompts use CLIP’s zero-shot capability to identify multiple anomaly types without labeled anomaly training data.
- Evaluation covers three surveillance benchmarks and live CCTV deployment at Chulalongkorn University, with stable real-time operation reported.
II. RELATED WORK
Related work includes reconstruction-, prediction-, and vision-language-based anomaly detection, while this paper emphasizes architectural simplification for real-time person-level detection.
- Reconstruction-based, prediction-based, and skeleton-trajectory methods detect anomalies through reconstruction error, predicted-frame deviations, or learned motion regularity.
- CLIP-based methods enable zero-shot or retraining-free anomaly detection through shared image-text embeddings and natural-language prompts.
- Existing vision-language approaches typically operate offline, whereas this work targets real-time deployment.
- The proposed contribution is architectural simplification: direct CLIP cosine similarity replaces multi-component density estimation for real-time person-level anomaly detection.
- YOLO v11n-pose provides person detection and seventeen-keypoint regression in one forward pass, balancing accuracy and speed for surveillance.
- The pipeline processes each frame through sequential person detection with pose estimation and CLIP-based semantic anomaly scoring.
A. Stage 1: Person Detection and Pose Estimation
Stage 1 detects people and estimates their poses with YOLO v11n-pose, then extracts padded person crops for downstream semantic scoring.
- Stage 1: Person Detection and Pose Estimation: YOLO v11n-pose simultaneously detects all persons and regresses their skeletal poses from each input frame.
- Stage 1: Person Detection and Pose Estimation: The model outputs each detected person’s bounding box, confidence, and seventeen COCO-topology keypoints.
- Stage 1: Person Detection and Pose Estimation: Keypoint visibility flags indicate whether each joint is visible, partially occluded, or not detected.
- Stage 1: Person Detection and Pose Estimation: Each person is cropped using bounding-box coordinates with a padding margin of δ = 10 pixels to preserve contextual information.
- Stage 1: Person Detection and Pose Estimation: The resulting crop set contains one crop per detection, with Nt denoting the number of detections at time t.
- Stage 1: Person Detection and Pose Estimation: On 640 × 480 CU Indoor Anomaly frames, δ = 10 pixels corresponds to approximately 1.6% of frame width and was selected empirically.
B. Stage 2: CLIP-Based Semantic Anomaly Scoring
Stage 2 encodes person crops and anomaly descriptions with CLIP, scores each person against the prompts, and aggregates scores at frame level.
- Stage 2: CLIP-Based Semantic Anomaly Scoring: Four textual prompts describe lying, falling, fighting, and sitting anomalies and are encoded into 512-dimensional text embeddings.
- Stage 2: CLIP-Based Semantic Anomaly Scoring: Text embeddings are computed once at initialization and cached for scoring.
- Stage 2: CLIP-Based Semantic Anomaly Scoring: Each person’s anomaly score is the maximum cosine similarity between its visual embedding and the prompt embeddings.
- Stage 2: CLIP-Based Semantic Anomaly Scoring: The frame-level anomaly score aggregates the scores of all detected persons.
C. Temporal Smoothing and Classification
Raw anomaly scores are Gaussian-smoothed over time, then compared with a decision threshold to classify frames as anomalous.
- Gaussian smoothing uses σ = 5 and a half-window of w = 15 to suppress transient score fluctuations.
- Frames are classified as anomalous when the smoothed score ˆSt meets or exceeds the threshold θ.
- θ∗= 0.7 is selected by sweeping thresholds from 0.1 to 0.9 on a validation partition to maximize AUROC.
IV. EXPERIMENTAL SETUP
The evaluation covers two established video-anomaly benchmarks and a custom indoor dataset, using a fine-tuned YOLO pose detector and established anomaly-detection metrics.
- Three datasets are evaluated: CUHK Avenue, ShanghaiTech Campus, and CU Indoor Anomaly from Chulalongkorn University.CU Indoor Anomaly contains 40 videos and frame-level annotations for falling, lying, sitting on the floor, and fighting.
- YOLO v11n-pose is fine-tuned on 2,864 training images from CU Indoor Anomaly for 500 epochs on an NVIDIA Titan XP GPU.The training images were augmented from 1,154 manually labeled frames.
- Frame-level AUROC is the primary evaluation metric, while YOLO detection additionally reports precision, recall, mAP@.5, and mAP@.5:.95.
- The proposed pipeline combines YOLO v11n-pose person detection and skeletal keypoint extraction with CLIP-based semantic scoring.
V. RESULTS AND ANALYSIS
The proposed framework matches the multi-feature baseline on CUHK Avenue, performs lower on ShanghaiTech Campus, and exceeds the baseline on CU Indoor Anomaly.
- 89.26% frame-level AUROC on CUHK Avenue matches the multi-feature baseline using FlowNet2, AlphaPose, and GMM/kNN scoring.
- 70.26% frame-level AUROC on ShanghaiTech Campus reflects diverse scenes and anomaly types outside the four fixed text prompts.
- 84.13% frame-level AUROC on CU Indoor Anomaly surpasses the baseline by roughly two percentage points.
- The method outperforms DSTN on Avenue by 2.86 percentage points while operating at substantially higher throughput.
B. Qualitative Detection Results
Qualitative outputs visualize normal and anomalous detections, while throughput analysis shows faster processing but persistent live-deployment overheads.
- Qualitative Detection Results: Normal persons receive green bounding boxes, whereas detected anomalies receive red boxes with YOLO confidence and CLIP scores.
- YOLO Detection Performance: 91% precision, 97% recall, and 92% mAP@.5 are achieved by YOLO v11n-pose on the CU Indoor Anomaly test set.The evaluation covers 237 images containing 360 annotated person instances.
- Processing Throughput: 51.39 FPS end-to-end throughput represents a 3.36× speedup over the baseline after removing its intermediate processing modules.
- Processing Throughput: Live deployment throughput includes overhead from frame decoding, CPU-to-GPU crop transfers, variable CLIP batch sizes, and Python/OpenCV rendering.
VI. DISCUSSION
The framework’s main strength is architectural simplicity: direct CLIP scoring removes several pipeline components and substantially increases throughput. Prompt-based scoring also supports retraining-free adaptation, while the method remains bounded to person-level anomalies in controlled indoor settings.
- Architectural simplicity: 51.39 FPS versus 15.32 FPS results from removing optical flow, standalone pose estimation, and density-model fitting.The simplified deployment uses one GPU and two model checkpoints.
- Prompt-based adaptation: Updating the prompt set adapts the system to a new surveillance environment without retraining.The fixed prompts also support operator inspection and modification without machine-learning expertise.
- Live deployment: Five live CCTV feeds operated stably at approximately 10 FPS per feed with capture, overlays, and concurrent anomaly-score graph updates.The deployment was conducted in the elevator area of a Chulalongkorn University building.
- Scope and limitations: The system is limited to person-level anomalies in controlled indoor settings, excluding non-person events and open-set categories outside the four fixed prompts.Production deployment would require complementary detection modules for these cases.
VII. CONCLUSION
The paper presents a YOLO v11n-pose and CLIP ViT-B/32 framework that replaces optical-flow and density-estimation components with direct text-image similarity. It reports a 3.36× throughput gain, benchmark AUROC performance, and stable live-feed operation, while acknowledging prompt and environmental scope limits.
- VII. CONCLUSION: The two-stage framework couples YOLO v11n-pose with CLIP ViT-B/32, replacing optical flow and density estimation.YOLO handles detection, while CLIP provides semantic anomaly scoring.
- VII. CONCLUSION: 3.36× throughput gain reaches 51.39 FPS while maintaining comparable AUROC on established benchmarks and improving the targeted indoor dataset to 84.13%.The reported improvement concerns the proposed direct text-image similarity scoring mechanism.
- VII. CONCLUSION: Stable operation above 10 FPS per feed was confirmed on live CCTV feeds at Chulalongkorn University.This result concerns deployment on live surveillance feeds.
- VII. CONCLUSION: The fixed four-prompt design constrains detectable anomaly categories, and indoor detector fine-tuning may reduce generalization to markedly different environments.ShanghaiTech Campus achieved a lower AUROC of 70.26% where some anomalies fell outside prompt coverage.
- VII. CONCLUSION: Future work proposes learnable prompt expansion, illumination-robust preprocessing, and a unified model for resource-constrained edge devices.Learnable prompt expansion is intended to broaden anomaly coverage.