Source-linked AI summary
Rethinking the Faster R-CNN Architecture for Temporal Action Localization
Yu-Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A. Ross, Jia Deng, Rahul Sukthankar
TL;DR
Temporal action localization must identify actions and their boundaries in untrimmed videos, where action durations vary widely and surrounding context matters. TAL-Net adapts Faster R-CNN with receptive-field alignment, explicit temporal context, and late motion fusion, achieving state-of-the-art proposal and localization performance on THUMOS’14 and competitive ActivityNet performance.
Problem
Temporal action localization must identify action classes and temporal boundaries in untrimmed videos, unlike classification on trimmed clips, while handling varied durations and relevant surrounding context.
Method
TAL-Net adapts Faster R-CNN using multi-scale receptive-field alignment, extended temporal context, and late fusion of multi-stream features.
Results
TAL-Net achieves state-of-the-art action proposal and localization performance on the THUMOS’14 detection benchmark and competitive performance on ActivityNet.
Takeaways & Limitations
TAL-Net’s design addresses receptive-field alignment, context extraction, and feature-fusion shortcomings in Faster R-CNN-based temporal action localization.
Abstract
from arXiv · showhide
We propose TAL-Net, an improved approach to temporal action localization in video that is inspired by the Faster R-CNN object detection framework. TAL-Net addresses three key shortcomings of existing approaches: (1) we improve receptive field alignment using a multi-scale architecture that can accommodate extreme variation in action durations; (2) we better exploit the temporal context of actions for both proposal generation and action classification by appropriately extending receptive fields; and (3) we explicitly consider multi-stream feature fusion and demonstrate that fusing motion late is important. We achieve state-of-the-art performance for both action proposal and localization on THUMOS'14 detection benchmark and competitive performance on ActivityNet challenge.
1. Introduction
Temporal action localization extends action recognition from trimmed clips to untrimmed videos by identifying action classes and their temporal boundaries. TAL-Net adapts Faster R-CNN to address duration variation, temporal context, and multi-stream fusion.
- Motivation: Temporal action localization detects action classes and start–end times in untrimmed videos containing irrelevant background activity.This contrasts with conventional action classification on temporally trimmed clips.
- Faster R-CNN framing: Region-based temporal localization first generates class-agnostic segment proposals and then classifies each proposal.This mirrors the proposal-plus-classification paradigm used in object detection.
- Architectural challenges: Faster R-CNN faces action-localization challenges from extreme action-duration variation and misalignment between receptive fields and anchor spans.TAL-Net addresses alignment with a multi-tower network and dilated temporal convolutions.
- Architectural challenges: Temporal context before and after actions is explicitly incorporated for proposal generation and action classification.The proposed receptive-field extensions target contextual information surrounding each action.
- Contributions: TAL-Net proposes late fusion of RGB and optical-flow features and reports state-of-the-art action proposal and localization performance on THUMOS’14.It also reports competitive performance on ActivityNet.
2. Related Work
Related work progresses from classification and sliding-window methods toward proposal-plus-classification architectures for temporal action localization. Faster R-CNN-based methods are powerful, but the paper argues that applying them naively to temporal localization leaves important issues unresolved.
- Action recognition: Conventional action recognition classifies temporally trimmed clips, limiting applicability to untrimmed real-world videos with irrelevant backgrounds.Temporal action localization instead targets long videos and action boundaries.
- Temporal action localization: Early localization methods use temporal sliding windows, pooled features, and SVM classifiers, with some introducing multi-scale pooling.These approaches classify actions within predefined windows.
- Temporal action localization: Frame-wise and snippet-wise methods infer action boundaries from labels while using temporal convolutions, recurrent networks, or separate contextual reasoning stages.Their central challenge is temporal contextual reasoning for individual labels.
- Proposal-based methods: Many recent methods adopt a two-stage framework that generates sparse class-agnostic segment proposals before classifying action categories.Some emphasize proposal quality, while others focus on action classification.
- Faster R-CNN-based methods: Faster R-CNN uses end-to-end trainable proposal and classification networks with boundary regression in both stages, but naive temporal adaptation may suffer from several issues.R-C3D is identified as a close Faster R-CNN-based example.
- Other approaches: Other related approaches include single-shot detectors, reinforcement learning, weakly supervised localization, and spatio-temporal localization outside this paper’s scope.These methods represent distinct problem formulations or settings.
3. Faster R-CNN
Faster R-CNN detects objects through proposal generation followed by classification and boundary regression. Its temporal counterpart replaces spatial regions with one-dimensional temporal segments and uses analogous proposal and classification stages.
- Object detection: Faster R-CNN processes an image with a 2D ConvNet, then generates class-agnostic region proposals from scale-varying anchor boxes.A Region Proposal Network classifies anchors and regresses their boundaries.
- Temporal adaptation: Temporal action localization is the 1D counterpart of object detection, replacing spatial bounding boxes with segments defined by start and end times.The analogy motivates adapting Faster R-CNN to video.
- Object detection: Each region proposal is pooled into a fixed-size feature map before a DNN predicts object classes and regresses class-specific boundaries.This second stage uses RoI pooling.
- Temporal adaptation: The temporal pipeline extracts a 1D feature map, classifies scale-varying anchor segments, regresses their boundaries, and returns sparse class-agnostic proposals.A Segment Proposal Network performs the first stage.
- Temporal adaptation: For each temporal proposal, SoI pooling feeds a DNN that predicts action classes and further regresses segment boundaries.This constitutes the second stage of the temporal Faster R-CNN pipeline.
4. TAL-Net
TAL-Net adapts Faster R-CNN to temporal action localization with aligned multi-scale receptive fields, explicit temporal context, and late two-stream fusion. Its Segment Proposal Network uses scale-specific temporal towers and dilated convolutions, while context-aware proposal and classification stages extend beyond each action span.
- 4. TAL-Net: TAL-Net introduces three architectural changes: receptive-field alignment, temporal-context extraction, and late multi-stream feature fusion.These changes target the domain-specific shortcomings of applying Faster R-CNN to temporal action localization.
- 4.1. Receptive Field Alignment: Action durations vary from less than a second to more than a minute, making a shared receptive field unsuitable for anchors spanning widely different temporal scales.A receptive field that is too small misses information for long anchors, while one that is too large may dilute short-action features.
- 4.1. Receptive Field Alignment: Each anchor scale receives a dedicated temporal ConvNet whose receptive field is designed to coincide with the anchor span.The multi-tower Segment Proposal Network uses parallel 1 × 1 layers for anchor classification and boundary regression.
- 4.1. Receptive Field Alignment: Dilated temporal convolutions control receptive-field size without loss of resolution, using two dilated layers and scale-dependent dilation rates.For target size s, the stated rates are r1 = s/6 and r2 = (s/6) × 2, with max pooling before the first convolution.
- 4.2. Context Feature Extraction: TAL-Net extends proposal and classification receptive fields to include context segments of length s/2 before and after an anchor or proposal.Proposal generation doubles the dilation rates, while SoI pooling covers the proposal plus both adjacent context regions.
- 4.3. Late Feature Fusion: The two-stream Faster R-CNN variant processes RGB and optical-flow features separately and averages logits late; experiments show late fusion outperforms early fusion.The same late-fusion principle is applied to proposal generation and action classification or boundary regression.
5. Experiments
Experiments evaluate TAL-Net on THUMOS’14 and ActivityNet using proposal recall, localization mAP, ablations, qualitative examples, and implementation details. Results support gains from context features and late fusion, with strongest reported localization performance on THUMOS’14.
- Dataset: THUMOS’14 contains 20 sports action classes, with 200 untrimmed validation videos for training and 213 test videos averaging over three minutes.The validation and test sets contain 3,007 and 3,358 action instances, respectively.
- Evaluation: The evaluation measures proposal Average Recall across Average Number of Proposals per Video and localization mean Average Precision across temporal IoU thresholds.Average Recall averages recall over temporal IoU thresholds from 0.5 to 1 in steps of 0.05.
- Ablations: Context features increase proposal Average Recall on both RGB and flow streams and improve localization mean Average Precision at nearly all proposal counts on both streams.The experiments evaluate context features separately for proposal generation and action classification.
- Ablations: Late fusion outperforms early fusion except at temporal IoU 0.1, while fused features outperform either single stream.The flow-based feature also outperforms the RGB-based feature in the reported comparison.
- State-of-the-art comparisons: 11.8% mAP is TAL-Net’s reported advantage over Gao et al. at temporal IoU 0.5 on THUMOS’14, with 42.8% versus 31.0%.TAL-Net achieves the highest mAP when the temporal IoU threshold exceeds 0.2.
- State-of-the-art comparisons: 38.23% versus 36.44% is TAL-Net’s localization mAP comparison with Dai et al. at temporal IoU 0.5 on ActivityNet v1.3.The corresponding comparison with Xu et al. is 38.23% versus 26.80%.
6. Conclusion
The conclusion presents TAL-Net as a Faster R-CNN-inspired architecture for temporal action localization. It attributes the approach’s results to receptive-field alignment, context extraction, and late feature fusion.
- Contributions: TAL-Net introduces three architectural changes addressing receptive field alignment, context feature extraction, and late feature fusion.The approach is designed for temporal action localization in video and is inspired by Faster R-CNN.
- Results: TAL-Net achieves state-of-the-art performance for action proposal and localization on THUMOS’14 and competitive performance on the ActivityNet challenge.
- Acknowledgement: The paper acknowledges assistance with the I3D models and optical flow.
A.1. Training Strategy
Training jointly optimizes proposal generation and action classification with classification and regression losses. Labels, sampling, and regression targets are defined relative to anchors or proposals.
- Loss: Proposal generation and action classification share a multi-task loss combining classification and regression objectives.Both stages are jointly trained with equal loss weighting, using λ = 1.
- Loss: Proposal classification uses binary labels, whereas action classification uses C action classes plus a background class.The classification loss is cross-entropy.
- Regression: Regression predicts center and length offsets relative to anchors or proposals using a smooth L1 loss.The indicator function excludes background anchors and proposals from regression loss computation.
- Label assignment: Proposal anchors are positive above tIoU 0.7, negative below 0.3 against all ground-truth segments, and ground-truth segments are forced to have a positive match.
- Optimization: Mini-batches contain examples from one video, with sizes 256 for proposal generation and 64 for action classification.Positive fractions are 0.5 and 0.25, respectively, and Adam uses learning rate 0.0001.
A.2. Additional Qualitative Results
Additional qualitative examples show that TAL-Net localizes actions successfully in most cases, while errors arise from boundaries, classification, indistinguishable motion, small objects, and occlusion.
- Qualitative results: TAL-Net successfully localizes actions in most qualitative examples.
- Failure cases: Observed failures include inaccurate boundaries, misclassified actions, false positives from indistinguishable body motions, and false negatives from small objects or occlusion.
A.3. Benchmarks using InceptionV3
Using ImageNet-pre-trained InceptionV3 features enables an apples-to-apples comparison with Zhao et al. TAL-Net outperforms that result by 7.7% in action-localization mAP on THUMOS’14.
- 7.7% higher action-localization mAP than Zhao et al. is achieved on THUMOS’14.The comparison uses ImageNet-pre-trained InceptionV3 features.
A.4. Computational Cost
Test-time cost is broken down across optical-flow extraction, I3D feature extraction, and proposal/classification, with optical flow dominating CPU runtime. Qualitative examples show predicted and ground-truth segments against a temporal axis.
- Test-time runtime is divided into optical-flow extraction, I3D feature extraction, and proposal and classification.
- 239 ms per frame makes optical-flow extraction the computational bottleneck in the CPU experiments.The authors note that GPU devices could provide further speedup.
- Qualitative examples display sampled video frames, ground-truth and predicted action segments, class labels, and time in seconds.