Multi-Scale Latent Temporal Compression Attention and Video-Text Cross Attention Fusion for Video Action Detection

1 Introduction

• Video action detection must recognize and temporally localize predefined actions and abnormal events in long, untrimmed streams.The introduction notes applications including surveillance, autonomous driving, behavior analysis, and embodied intelligence.Source: §1 · p. 1
• MLTC-Attention compresses temporal features at multiple latent-space scales before attention to balance global context and local detail while reducing redundancy.A gate re-weights scales according to input content.Source: §1 · pp. 2–3
• VTCA-Fusion aligns video and category-text features through cross-attention at standard and latent levels for fine-grained alignment and complementary integration.The standard feature space captures explicit semantic correspondences, while the latent branch captures compressed cross-modal patterns.Source: §1 · pp. 2–3
• Experiments on four benchmarks demonstrate the effectiveness of the proposed modules.The framework is described as applicable to various video action detection tasks.Source: §1 · p. 3

2.1 Video Understanding Models

• Video representations progress from 3D CNNs through Transformers and self-supervised models to state-space approaches; multimodal work aligns vision with language and connects video encoders to large language models.The section also mentions CLIP-based video action recognition.Source: §2.1 · p. 3

2.2 Video Action Detection Models

• Temporal action detection spans boundary-matching and graph-based methods, multi-scale feature pyramids and pooling, and Transformer- and state-space-based temporal modeling.Recent work also explores cross-layer decoupling and boundary refinement.Source: §2.2 · p. 3

2.3 Video Anomaly Detection Models

• Video anomaly detection work includes MIL-based learning, graph and memory methods, CLIP transfer, text-prompt guidance, and modeling anomalies across temporal scales.The framework overview depicts VTCA-Fusion and MLTC-Attention processing temporal and categorical features for downstream tasks.Source: §2.3 · p. 3, Figure 2 · p. 4

3.1 Overview

• Video features and encoded category-text prompts are fused into a temporal sequence and similarity map, which feed MLTC-Attention and produce temporal and class scores.The video encoder extracts frame features, while enhanced category prompts are encoded and projected into the video feature space.Source: §3.1 · p. 4

3.2 Video-Text Cross Attention Fusion Module

• VTCA-Fusion uses video frames as queries and enhanced text features as keys and values, combining standard-level semantic detail with compressed latent-level interaction.The two attention outputs are concatenated, projected, and added to the video features to form the fused temporal sequence.Source: §3.2 · p. 5

3.3 Multi-Scale Latent Temporal Compression Attention Block

• MLTC-Attention compresses latent keys and values at multiple temporal scales, attends to compressed and top-k critical segments, then gates these outputs with global self-attention.Compression reduces each branch’s cost from O(τ^2) to O(τ^2/σi), while the global branch retains the full receptive field.Source: §3.3 · pp. 5–7

Loss of Video Anomaly Detection.

• For video anomaly detection, video-level labels are broadcast across frames, while the loss combines binary anomaly classification, top-k low-scoring abnormal-video frames treated as normal, and category classification.The additional normal-instance loss addresses noise from abnormal videos containing normal frames.Source: Loss of Video Anomaly Detection · p. 7

Loss of Temporal Action Detection.

• For temporal action detection, training combines Focal Loss for action classification with Distance-IoU for temporal segment regression, weighted by λ, and inference applies Soft Non-Maximum Suppression.The classification and regression labels correspond to action categories and temporal segments, respectively.Source: Loss of Temporal Action Detection · pp. 7–8

4.1 Datasets and Evaluation Metrics

• On UCF-Crime and XD-Violence, frame-level AUC and AP are reported; THUMOS14 and ActivityNet mAP is averaged over multiple tIoU thresholds.The evaluation uses UCF-Crime and XD-Violence for anomaly detection, and THUMOS14 and ActivityNet for temporal action detection.Source: §4.1 · p. 8

4.2 Implementation Details

• The model uses frozen pretrained video and text encoders, trains projection layers, prompts, both proposed modules and the task head, and runs for 20 epochs.Training uses AdamW, a learning rate of 1e-5, batch size 64, and 512-frame video inputs on one NVIDIA L40S GPU.Source: §4.2 · p. 8

4.3 Evaluation and Comparison

• The method achieves the highest reported AUC on UCF-Crime and AP on XD-Violence without audio, and leads across tIoU thresholds on THUMOS14 and ActivityNet-1.3.It also has the highest throughput with the fewest FLOPs among compared methods while being the most accurate.Source: §4.3 · pp. 8–10, Table 1 · p. 9, Table 2 · p. 9

4.4 Qualitative Analyses

• Qualitative predictions are more precise than the baseline on both tasks, while category-wise attention activates on anomaly intervals and favors “Normal” for normal videos.The visualizations show anomaly scores against ground-truth intervals and predicted action categories over temporal segments.Source: §4.4 · pp. 10–11, Figure 3 · p. 10, Figure 4 · p. 10

4.5 Ablation Studies

• Ablations show that replacing MLTC-Attention with standard attention, or removing VTCA-Fusion or latent components, lowers performance; compression scales balance fine detail and redundancy reduction.Learnable prompts achieve the best results; removing text or using random, generic, or class-only prompts performs worse.Source: §4.5 · pp. 11–12

• The methods consistently improve accuracy and achieve superior overall performance on temporal action detection and video anomaly detection benchmarks.Source: §5 · p. 12
• MLTC-Attention uses multi-scale temporal latent-space compression to capture dynamic features while reducing computational redundancy and inefficiency.Source: §5 · p. 12
• VTCA-Fusion applies cross-attention between video content and enhanced multi-category text for precise alignment and complementary integration of visual and textual features.Source: §5 · p. 12