Real-Time Video Anomaly Detection Using YOLO Pose Estimation and CLIP-Based Semantic Scoring

• Human attention fatigue in multi-feed CCTV monitoring motivates automated detection of falls, altercations, and abnormal postures.Source: §I · p. 1
• Prior multi-feature pipelines combining AlphaPose, FlowNet2, CLIP, and GMM/kNN scoring reach approximately 15 FPS, limiting throughput.Source: §I · p. 1
• YOLO v11n-pose performs detection and pose estimation while CLIP ViT-B/32 provides semantic anomaly scoring in the proposed two-stage framework.Source: §I · p. 1
• 3.36× throughput gain over the prior baseline is reported for the streamlined pipeline using direct CLIP semantic scoring instead of multi-feature density estimation.Source: §I · p. 1
• Text prompts use CLIP’s zero-shot capability to identify multiple anomaly types without labeled anomaly training data.Source: §I · p. 1
• Evaluation covers three surveillance benchmarks and live CCTV deployment at Chulalongkorn University, with stable real-time operation reported.Source: §I · p. 1

• Reconstruction-based, prediction-based, and skeleton-trajectory methods detect anomalies through reconstruction error, predicted-frame deviations, or learned motion regularity.Source: §II · p. 1
• CLIP-based methods enable zero-shot or retraining-free anomaly detection through shared image-text embeddings and natural-language prompts.Source: §II · p. 1
• Existing vision-language approaches typically operate offline, whereas this work targets real-time deployment.Source: §II · p. 1
• The proposed contribution is architectural simplification: direct CLIP cosine similarity replaces multi-component density estimation for real-time person-level anomaly detection.Source: §II · p. 2
• YOLO v11n-pose provides person detection and seventeen-keypoint regression in one forward pass, balancing accuracy and speed for surveillance.Source: §II · p. 2
• The pipeline processes each frame through sequential person detection with pose estimation and CLIP-based semantic anomaly scoring.Source: §III · p. 2

Stage 1: Person Detection and Pose Estimation

• YOLO v11n-pose simultaneously detects all persons and regresses their skeletal poses from each input frame.Source: A. Stage 1: Person Detection and Pose… · p. 2
• The model outputs each detected person’s bounding box, confidence, and seventeen COCO-topology keypoints.Source: A. Stage 1: Person Detection and Pose… · p. 2
• Keypoint visibility flags indicate whether each joint is visible, partially occluded, or not detected.Source: A. Stage 1: Person Detection and Pose… · p. 2
• Each person is cropped using bounding-box coordinates with a padding margin of δ = 10 pixels to preserve contextual information.Source: A. Stage 1: Person Detection and Pose… · p. 2
• The resulting crop set contains one crop per detection, with Nt denoting the number of detections at time t.Source: A. Stage 1: Person Detection and Pose… · p. 2
• On 640 × 480 CU Indoor Anomaly frames, δ = 10 pixels corresponds to approximately 1.6% of frame width and was selected empirically.Source: A. Stage 1: Person Detection and Pose… · p. 2

Stage 2: CLIP-Based Semantic Anomaly Scoring

• Four textual prompts describe lying, falling, fighting, and sitting anomalies and are encoded into 512-dimensional text embeddings.Source: B. Stage 2: CLIP-Based Semantic… · p. 2
• Text embeddings are computed once at initialization and cached for scoring.Source: B. Stage 2: CLIP-Based Semantic… · p. 2
• Each person’s anomaly score is the maximum cosine similarity between its visual embedding and the prompt embeddings.Source: B. Stage 2: CLIP-Based Semantic… · p. 2
• The frame-level anomaly score aggregates the scores of all detected persons.Source: B. Stage 2: CLIP-Based Semantic… · p. 2

• Gaussian smoothing uses σ = 5 and a half-window of w = 15 to suppress transient score fluctuations.Source: C. Temporal Smoothing and Classification · p. 2
• Frames are classified as anomalous when the smoothed score ˆSt meets or exceeds the threshold θ.Source: C. Temporal Smoothing and Classification · p. 2
• θ∗= 0.7 is selected by sweeping thresholds from 0.1 to 0.9 on a validation partition to maximize AUROC.Source: C. Temporal Smoothing and Classification · p. 2

• Three datasets are evaluated: CUHK Avenue, ShanghaiTech Campus, and CU Indoor Anomaly from Chulalongkorn University.CU Indoor Anomaly contains 40 videos and frame-level annotations for falling, lying, sitting on the floor, and fighting.Source: A. Datasets · p. 2
• YOLO v11n-pose is fine-tuned on 2,864 training images from CU Indoor Anomaly for 500 epochs on an NVIDIA Titan XP GPU.The training images were augmented from 1,154 manually labeled frames.Source: B. Implementation Details · p. 2
• Frame-level AUROC is the primary evaluation metric, while YOLO detection additionally reports precision, recall, mAP@.5, and mAP@.5:.95.Source: C. Evaluation Protocol · p. 2
• The proposed pipeline combines YOLO v11n-pose person detection and skeletal keypoint extraction with CLIP-based semantic scoring.Source: Figure 1 · p. 3

• 89.26% frame-level AUROC on CUHK Avenue matches the multi-feature baseline using FlowNet2, AlphaPose, and GMM/kNN scoring.Source: A. Anomaly Detection Performance · p. 3
• 70.26% frame-level AUROC on ShanghaiTech Campus reflects diverse scenes and anomaly types outside the four fixed text prompts.Source: A. Anomaly Detection Performance · p. 3
• 84.13% frame-level AUROC on CU Indoor Anomaly surpasses the baseline by roughly two percentage points.Source: A. Anomaly Detection Performance · p. 3
• The method outperforms DSTN on Avenue by 2.86 percentage points while operating at substantially higher throughput.Source: A. Anomaly Detection Performance · p. 3

Qualitative Detection Results

• Normal persons receive green bounding boxes, whereas detected anomalies receive red boxes with YOLO confidence and CLIP scores.Source: B. Qualitative Detection Results · p. 3, Figure 2 · p. 4

YOLO Detection Performance

• 91% precision, 97% recall, and 92% mAP@.5 are achieved by YOLO v11n-pose on the CU Indoor Anomaly test set.The evaluation covers 237 images containing 360 annotated person instances.Source: C. YOLO Detection Performance · p. 3

Processing Throughput

• 51.39 FPS end-to-end throughput represents a 3.36× speedup over the baseline after removing its intermediate processing modules.Source: D. Processing Throughput · pp. 3–4
• Live deployment throughput includes overhead from frame decoding, CPU-to-GPU crop transfers, variable CLIP batch sizes, and Python/OpenCV rendering.Source: D. Processing Throughput · p. 4

Architectural simplicity

• 51.39 FPS versus 15.32 FPS results from removing optical flow, standalone pose estimation, and density-model fitting.The simplified deployment uses one GPU and two model checkpoints.Source: §VI · p. 4

Prompt-based adaptation

• Updating the prompt set adapts the system to a new surveillance environment without retraining.The fixed prompts also support operator inspection and modification without machine-learning expertise.Source: §VI · p. 4

Live deployment

• Five live CCTV feeds operated stably at approximately 10 FPS per feed with capture, overlays, and concurrent anomaly-score graph updates.The deployment was conducted in the elevator area of a Chulalongkorn University building.Source: §VI · p. 4

Scope and limitations

• The system is limited to person-level anomalies in controlled indoor settings, excluding non-person events and open-set categories outside the four fixed prompts.Production deployment would require complementary detection modules for these cases.Source: §VI · p. 4

VII. CONCLUSION

• The two-stage framework couples YOLO v11n-pose with CLIP ViT-B/32, replacing optical flow and density estimation.YOLO handles detection, while CLIP provides semantic anomaly scoring.Source: §VII · p. 4
• 3.36× throughput gain reaches 51.39 FPS while maintaining comparable AUROC on established benchmarks and improving the targeted indoor dataset to 84.13%.The reported improvement concerns the proposed direct text-image similarity scoring mechanism.Source: §VII · pp. 4–5
• Stable operation above 10 FPS per feed was confirmed on live CCTV feeds at Chulalongkorn University.This result concerns deployment on live surveillance feeds.Source: §VII · p. 5
• The fixed four-prompt design constrains detectable anomaly categories, and indoor detector fine-tuning may reduce generalization to markedly different environments.ShanghaiTech Campus achieved a lower AUROC of 70.26% where some anomalies fell outside prompt coverage.Source: §VII · p. 5
• Future work proposes learnable prompt expansion, illumination-robust preprocessing, and a unified model for resource-constrained edge devices.Learnable prompt expansion is intended to broaden anomaly coverage.Source: §VII · p. 5