Source-linked AI summary
You Only Watch Once: A Unified CNN Architecture for Real-Time Spatiotemporal Action Localization
Okan Köpüklü, Xiangyu Wei, Gerhard Rigoll
TL;DR
Existing two-stage pipelines for spatiotemporal action localization have three main shortcomings. YOWO addresses this with a unified single-stage architecture, achieving state-of-the-art results on UCF101-24 and J-HMDB-21, competitive AVA results, and real-time operation.
Problem
Existing two-stage pipelines for spatiotemporal action localization have three main shortcomings.
Method
YOWO is a unified single-stage framework using a 3D-CNN to process previous frames in a causal architecture.
Results
YOWO outperforms other state-of-the-art results on UCF101-24 and J-HMDB-21, achieves competitive AVA results, and operates in real time.
Takeaways & Limitations
The unified architecture supports competitive spatiotemporal action localization across UCF101-24, J-HMDB-21, and AVA while operating in real time.
Takeaways & Limitations
Downsampling can hinder motion-pattern capture, while long sequences may contain several unrelated frames.
Abstract
from arXiv · showhide
Spatiotemporal action localization requires the incorporation of two sources of information into the designed architecture: (1) temporal information from the previous frames and (2) spatial information from the key frame. Current state-of-the-art approaches usually extract these information with separate networks and use an extra mechanism for fusion to get detections. In this work, we present YOWO, a unified CNN architecture for real-time spatiotemporal action localization in video streams. YOWO is a single-stage architecture with two branches to extract temporal and spatial information concurrently and predict bounding boxes and action probabilities directly from video clips in one evaluation. Since the whole architecture is unified, it can be optimized end-to-end. The YOWO architecture is fast providing 34 frames-per-second on 16-frames input clips and 62 frames-per-second on 8-frames input clips, which is currently the fastest state-of-the-art architecture on spatiotemporal action localization task. Remarkably, YOWO outperforms the previous state-of-the art results on J-HMDB-21 and UCF101-24 with an impressive improvement of ~3% and ~12%, respectively. Moreover, YOWO is the first and only single-stage architecture that provides competitive results on AVA dataset. We make our code and pretrained models publicly available.
1. Introduction
Spatiotemporal action localization must combine temporal information from previous frames with spatial information from a key frame. YOWO addresses limitations of two-stage pipelines with a unified, causal, single-stage architecture for efficient online localization.
- Problem: Two-stage action-localization pipelines generate complex action tubes, depend on proposal quality, neglect background context, and require separately trained stages.These limitations increase computational cost and may prevent global optimization.
- Problem: Temporal information from previous frames is necessary because action classification cannot always be determined from the key frame alone.The paper illustrates this distinction with sitting-versus-standing recognition.
- Method: YOWO uses two concurrent branches: a 2D-CNN extracts key-frame spatial features, while a 3D-CNN models spatiotemporal features from previous frames.The branches are fused with channel fusion and attention before frame-level detections and action tubes are produced.
- Method: YOWO is a causal architecture that operates online on incoming video streams and can be adapted to modalities such as optical flow or depth.The implementation uses RGB, while the branch design is not restricted to RGB.
- Results: 6.9% and 1.3% frame-mAP increases result from averaging long-term 3D features around the key frame during inference.The long-term feature bank uses non-overlapping 8-frame clips to supplement short input clips.
- Results: 3.3% and 12.2% frame-mAP improvements are reported on J-HMDB-21 and UCF101-24, while runtime reaches 34 fps with 16-frame clips and 62 fps with 8-frame clips.The paper presents these results as establishing new state-of-the-art performance and fastest state-of-the-art runtime.
2. Related Work
Prior work combines spatial and temporal video information through recurrent, 3D-CNN, two-stream, proposal-based, or attention-based approaches, but often faces computational or architectural limitations. YOWO instead introduces a unified framework that jointly uses key-frame and clip features for spatiotemporal action localization.
- Video feature extraction: Two-stream methods separately extract spatial and temporal features, commonly using optical flow, whose computation is time-consuming.This motivates alternatives that avoid explicit optical-flow processing.
- Video feature extraction: 3D-CNNs learn spatial and temporal features simultaneously but generally require more parameters than 2D-CNNs, making them computationally expensive.Resource-efficient variants and hybrid 2D/3D approaches were explored to reduce this burden.
- Spatiotemporal action localization: Two-stage architectures are difficult to implement for real-time tasks, while jointly training their proposal and classification networks may not reach the global optimum.One-stage detectors such as YOLO and SSD were developed to simplify detection and improve real-time performance.
- Spatiotemporal action localization: Video action-localization systems include two-stage human detection and action-tube linking, clip-tube proposal methods, and dense 3D architectures.Several of these approaches rely on region proposals or incur high computational complexity.
- Attention modules: Attention mechanisms model spatial or channel relationships, with channel attention emphasizing meaningful channels and weakening others.Non-local blocks extend attention to dependencies across video frames.
- YOWO: YOWO unifies clip and key-frame processing by combining 3D and 2D features, uses channel attention for fusion, and detects actions in the key frame.The framework is designed for spatiotemporal action localization and avoids complex optical-flow computation.
3. Methodology
YOWO is a unified architecture with concurrent 3D-CNN and 2D-CNN branches for temporal clip and spatial key-frame features. These features are aligned, aggregated through CFAM, and passed to a single-stage bounding-box prediction head that can be trained end-to-end.
- Architecture: YOWO comprises a 3D-CNN branch, a 2D-CNN branch, CFAM, and a bounding-box prediction component.The branches extract complementary video and key-frame representations concurrently.
- Feature alignment: The 3D and 2D branches are aligned to compatible spatial dimensions and their outputs are sent to CFAM for feature aggregation.The 3D feature volume is squeezed after its depth dimension is reduced to 1.
- Architecture flexibility: Both CNN branches can be replaced by arbitrary CNN architectures, making YOWO flexible while retaining a unified end-to-end training design.Although it has two branches, the parameters can be updated jointly.
Feature aggregation: Channel Fusion and Attention Mechanism (CFAM)
CFAM fuses the aligned 2D and 3D feature maps and models dependencies among channels through Gram-matrix-based attention. The resulting representation combines motion and appearance information before bounding-box prediction and subsequent tube construction.
- Channel fusion: CFAM first aligns and fuses the two branch feature maps before applying channel-wise attention.Its output is then processed by additional convolutional layers for final predictions.
- Attention computation: The attention map is computed from inter-channel correlations obtained by vectorizing each feature channel and forming a Gram matrix.Each Gram-matrix element represents the inner product between two vectorized feature maps.
- Attention computation: The attention weights measure each channel’s impact on another channel, then reweight the features and combine them with the original input using a trainable scalar.The resulting channel features model long-range semantic dependencies.
- Ablation and performance: The convolutional layers at the beginning and end of CFAM are important for mixing features from different backbones; removing them only marginally improves performance.The paper reports that these layers account for a significant performance effect.
- Feature representation: CFAM allocates greater weights to correlated channels, emphasizing contextual relationships and improving feature discriminability.The fused feature map encodes both motion and appearance information.
- Prediction and tube construction: The fused representation feeds a YOLO-style prediction head, while linked frame-level detections and Viterbi decoding generate action tubes.The linking score includes an additional term based on confidence scores in successive frames, which improves video detection performance in experiments.
4. Experiments
Experiments evaluate YOWO on UCF101-24, J-HMDB-21, and AVA using established detection metrics and ablations of its 2D, 3D, and CFAM components. Results show complementary spatial and temporal features, improved performance from fusion and attention, and sensitivity to clip length and downsampling.
- Datasets and metrics: YOWO is evaluated on UCF101-24, J-HMDB-21, and AVA using the datasets’ official evaluation protocols.The experiments compare the method with state-of-the-art approaches.
- Datasets and metrics: Frame-mAP measures per-frame detection precision-recall, whereas video-mAP evaluates action tubes using temporal IoU and correct action labels.AVA is evaluated only with frame-mAP at IoU 0.5.
- Ablation study: Neither a standalone 2D-CNN nor a standalone 3D-CNN adequately solves spatiotemporal localization, while their combination exploits complementary information.The 2D network captures finer spatial features, whereas the 3D network focuses more on motion.
- Ablation study: 2D networks localize better, 3D networks classify better, and CFAM improves both localization and classification on UCF101-24 and J-HMDB-21.Activation visualizations likewise show 3D features concentrated on motion and 2D features covering fine spatial information across people.
- Input configuration: 16-frame inputs outperform 8-frame inputs, while increased downsampling worsens performance because it hinders motion-pattern capture.Longer clips provide more temporal information, but overly long sequences may contain unrelated frames for quick motions.
- Comparison with prior work: YOWO improves frame-mAP by 3.3% on J-HMDB-21 and achieves 80.4% frame-mAP on UCF101-24 at IoU 0.5.It also reports competitive video-mAP results and is described as a single-stage architecture with competitive AVA performance.
5. Conclusion
YOWO is a unified architecture for spatiotemporal action localization that processes temporal context and key-frame spatial information in parallel. It jointly optimizes detection and classification, achieving strong results across three datasets while operating causally in real time.
- YOWO models spatiotemporal context from successive frames while extracting fine spatial information from the key frame in parallel.
- Channel fusion and attention aggregate temporal and spatial information within the architecture.
- Separating neither human detection nor action classification enables joint-loss optimization of the whole network end to end.
- Comparative evaluations were conducted on UCF101-24, J-HMDB-21, and AVA, which have different characteristics.
- The approach outperforms state-of-the-art results on UCF101-24 and achieves competitive results on AVA.
- YOWO is causal, operates in real time, and can therefore be deployed on mobile devices.