Source-linked AI summary
Weakly Supervised Action Localization by Sparse Temporal Pooling Network
Phuc Nguyen, Ting Liu, Gautam Prasad, Bohyung Han
TL;DR
Temporal action localization in untrimmed videos typically requires temporal annotations, but this paper learns from video-level class labels alone. Its network selects sparse informative segments and combines attention with temporal class activations to produce action proposals, achieving state-of-the-art THUMOS14 performance and outstanding ActivityNet1.3 results. The approach depends on fixed I3D features pretrained on Kinetics without fine-tuning.
Problem
Temporal localization in untrimmed videos requires expensive temporal interval annotations, motivating competitive methods trained with minimal supervision.
Method
A deep network learns sparse attention-weighted segment representations using classification and sparsity losses, then uses T-CAMs and class-agnostic attention to localize actions.
Results
The method achieves state-of-the-art performance on THUMOS14 and reports weakly supervised temporal localization results on ActivityNet1.3 for the first time.
Takeaways & Limitations
Weakly supervised learning can produce temporal action localization on untrimmed videos using video-level class labels rather than temporal annotations.
Takeaways & Limitations
The network uses fixed I3D features pretrained on Kinetics without fine-tuning, so it may not be optimized for target classes and datasets.
Abstract
from arXiv · showhide
We propose a weakly supervised temporal action localization algorithm on untrimmed videos using convolutional neural networks. Our algorithm learns from video-level class labels and predicts temporal intervals of human actions with no requirement of temporal localization annotations. We design our network to identify a sparse subset of key segments associated with target actions in a video using an attention module and fuse the key segments through adaptive temporal pooling. Our loss function is comprised of two terms that minimize the video-level action classification error and enforce the sparsity of the segment selection. At inference time, we extract and score temporal proposals using temporal class activations and class-agnostic attentions to estimate the time intervals that correspond to target actions. The proposed algorithm attains state-of-the-art results on the THUMOS14 dataset and outstanding performance on ActivityNet1.3 even with its weak supervision.
1. Introduction
The paper targets temporal action localization in untrimmed videos without expensive temporal annotations. It proposes weakly supervised learning that selects sparse useful segments and combines class-specific activation maps with class-agnostic attention for localization.
- The method uses a two-stream input of RGB frames and optical flow for concurrent action classification and temporal localization.
- Temporal Class Activation Maps generate one-dimensional temporal proposals, while class-agnostic attentions help identify segments relevant to actions.
- Untrimmed videos contain irrelevant frames, while temporal action localization commonly requires expensive interval annotations.
- The proposed network learns from video-level class labels to select a sparse subset of segments for recognizing and localizing actions.
- The technique achieves state-of-the-art results on THUMOS14 and outstanding performance on ActivityNet1.3 under weak supervision.
2. Related Work
Prior video understanding work includes recognition and localization methods, but many localization systems rely on temporal or spatiotemporal supervision. Weakly supervised alternatives use attention weights or temporal subaction order, while this paper focuses on full-action intervals.
- Action recognition is commonly addressed with CNNs, two-stream networks, and 3D convolutional networks that learn video representations.
- Action localization differs from recognition because it detects temporal or spatiotemporal volumes containing target actions.
- Many localization methods rely on supervised learning with temporal or spatiotemporal annotations and use action proposals.
- UntrimmedNet uses temporal-softmax attention on precut segments and thresholds class-agnostic attention weights to generate proposals.
- Other weakly supervised methods exploit temporal subaction order to find sequential boundaries, whereas this approach extracts intervals of full actions.
- Trimmed recognition datasets differ from THUMOS14 and ActivityNet, whose untrimmed videos include background frames and temporal action annotations.
3. Proposed Algorithm
The proposed method learns sparse, class-agnostic segment selection from video-level labels, then combines attention and class-specific temporal activations to localize actions in untrimmed videos.
- Architecture and training: The network selects a sparse subset of representative video segments using learned importance weights and only video-level class labels.Its training objective combines video-level classification error with sparsity of the selected segments.
- Architecture and training: Uniformly sampled segment features pass through an attention module whose weights produce a video representation by attention-weighted temporal average pooling.The attention weights are class-agnostic and intended to identify segments relevant across actions of interest.
- Architecture and training: The loss combines video-level classification loss with an ℓ1 sparsity loss on attention weights, with β controlling their trade-off.Sigmoid outputs and the ℓ1 penalty encourage attention weights toward values close to 0 or 1.
- Temporal localization: Class-agnostic attentions measure generic actionness, whereas T-CAMs provide class-specific information for distinguishing target actions during localization.Some highly attended intervals may have low T-CAM values because they can correspond to other actions of interest.
- Implementation and scope: The method uses separate RGB and optical-flow recognition networks with I3D features pretrained on Kinetics and not fine-tuned on target datasets.This design supports the two-stream proposal generation procedure but may leave the representation mismatched to target classes.
- Temporal localization: At inference, relevant classes are identified from video-level scores, and class-specific T-CAMs are combined with attentions to form temporal proposals.Thresholded weighted T-CAM signals yield connected temporal components, which receive class scores and undergo class-wise non-maximum suppression.
4. Experiments
Experiments evaluate STPN on untrimmed THUMOS14 and ActivityNet1.3 videos using weak supervision, compare it with prior methods, and analyze architectural and modality choices. Results show strong localization performance, while ablations support sparse attention-weighted pooling and combining RGB with optical flow.
- Datasets and Evaluation Method: STPN is evaluated on THUMOS14 and ActivityNet1.3, both untrimmed benchmarks containing irrelevant frames and multiple actions.Training does not use temporal annotations; evaluation follows mean average precision at multiple IoU thresholds.
- Datasets and Evaluation Method: THUMOS14 includes long videos with multiple instances and highly variable action durations, making temporal localization challenging.The dataset uses 200 untrimmed validation videos for training without temporal annotations and 212 temporally annotated testing videos.
- Results: STPN achieves state-of-the-art weakly supervised performance on THUMOS14 and remains competitive with several fully supervised approaches.It also outperforms the compared weakly supervised methods and the UntrimmedNet and Hide-and-Seek variants in the reported setting.
- Results: STPN outperforms some fully supervised approaches on both the ActivityNet1.3 validation and testing sets.The paper notes that many leaderboard results are tuned for the ActivityNet Challenge and may not be directly comparable; these experiments provide a weakly supervised baseline.
- Results: Qualitative THUMOS14 results show localization despite similar appearance, little motion, multiple similar actions, and false positives partly associated with missing annotations or visually similar actions.The model can identify multiple action categories and temporal boundaries under these conditions.
- Ablation Study: Ablations find substantial gains from attention-weighted pooling and sparsity loss, while optical flow performs better than RGB alone and combining modalities improves performance.The ablation experiments are conducted on THUMOS14.
5. Conclusion
The paper presents a weakly supervised deep-learning approach for temporal action localization, using sparse segment weighting and proposal selection without temporal annotations. It achieves state-of-the-art performance on THUMOS14 and reports weakly supervised localization results on ActivityNet1.3 for the first time.
- The method classifies videos using a sparsely weighted mean of segment-level features, with sparse coefficients learned through a sparsity loss.
- One-dimensional action proposals are extracted and filtered for target classes to identify temporal action intervals.
- The approach achieves state-of-the-art performance on THUMOS14.
- The paper reports weakly supervised temporal action localization results on ActivityNet1.3 for the first time.