Source-linked AI summary
Two-Stream Consensus Network for Weakly-Supervised Temporal Action Localization
Yuanhao Zhai, Le Wang, Wei Tang, Qilin Zhang, Junsong Yuan, Gang Hua
TL;DR
W-TAL must localize actions from video-level labels despite lacking frame-level annotations, making false positives and imprecise boundaries difficult to control. TSCN addresses this with iterative pseudo-ground-truth supervision and attention normalization, outperforming prior W-TAL methods on two benchmarks and matching some fully supervised results.
Problem
W-TAL lacks frame-level annotations, making false positive proposals and precise action-instance boundaries difficult to obtain.
Method
TSCN iteratively updates frame-level pseudo ground truth from late-fusion RGB-flow attention and uses attention normalization to make predictions more binary.
Results
TSCN outperforms current state-of-the-art W-TAL methods on THUMOS14 and ActivityNet and achieves comparable results to some recent fully supervised methods.
Takeaways & Limitations
Iterative fusion-based supervision and attention normalization improve proposal precision and localization within the evaluated W-TAL benchmarks.
Abstract
from arXiv · showhide
Weakly-supervised Temporal Action Localization (W-TAL) aims to classify and localize all action instances in an untrimmed video under only video-level supervision. However, without frame-level annotations, it is challenging for W-TAL methods to identify false positive action proposals and generate action proposals with precise temporal boundaries. In this paper, we present a Two-Stream Consensus Network (TSCN) to simultaneously address these challenges. The proposed TSCN features an iterative refinement training method, where a frame-level pseudo ground truth is iteratively updated, and used to provide frame-level supervision for improved model training and false positive action proposal elimination. Furthermore, we propose a new attention normalization loss to encourage the predicted attention to act like a binary selection, and promote the precise localization of action instance boundaries. Experiments conducted on the THUMOS14 and ActivityNet datasets show that the proposed TSCN outperforms current state-of-the-art methods, and even achieves comparable results with some recent fully-supervised methods.
1 Introduction
W-TAL uses only video-level labels to localize and classify actions, but struggles with false positives and imprecise proposal boundaries. TSCN addresses these challenges through late-fusion-based iterative supervision and attention normalization.
- Motivation: W-TAL localizes and classifies action instances in untrimmed videos using only video-level categorical labels.This reduces data-collection effort compared with frame-level annotation.
- Challenges: Without frame-level annotations, prior methods may localize actions unrelated to the video label, such as detecting swimming from water alone.The paper identifies fine-grained supervision as necessary to guide learning.
- Challenges: Fixed attention thresholds create a trade-off: high values yield incomplete proposals, while low values increase false positives.The threshold-setting dilemma was rarely studied in prior work.
- TSCN: TSCN generates frame-level pseudo ground truth from weighted RGB-flow attention and iteratively uses it to supervise two-stream models.The method treats late fusion as a voting ensemble whose combined activations can suppress stream-specific false positives.
- TSCN: Attention normalization pushes attention toward binary selection, improving the quality of threshold-generated action proposals.The loss is presented as a complement to iterative refinement and is evaluated on THUMOS14 and ActivityNet.
- Results: TSCN significantly outperforms prior state-of-the-art W-TAL methods and achieves comparable results to some recent fully supervised methods.The reported experiments use the THUMOS14 and ActivityNet benchmarks.
2 Related Work
Related work spans two-stream action recognition, fully supervised localization, and weakly supervised localization. TSCN differs from prior refinement methods through late fusion, fusion-derived pseudo ground truth, and explicit attention normalization.
- Action Recognition: Two-Stream Convolutional Networks separately model RGB appearance and optical-flow motion, then reconcile outputs through late fusion.This two-modality design motivates the fusion strategy used by TSCN.
- Fully-Supervised TAL: Fully supervised temporal localization requires frame-level annotations, with datasets including THUMOS, ActivityNet, and Charades.This setting provides stronger supervision than W-TAL.
- Weakly-Supervised TAL: W-TAL requires only video-level supervision during training and therefore reduces data-annotation effort.Prior methods include Hide-and-Seek, UntrimmedNet, and Sparse Temporal Pooling Network.
- Comparison with RefineLoc: RefineLoc uses iterative refinement, but TSCN uses late fusion whereas RefineLoc uses early fusion.The comparison distinguishes the two methods’ fusion architectures.
- Comparison with RefineLoc: TSCN derives pseudo ground truth from two-stream late-fusion attention, while RefineLoc expands previous localization results.The paper states that fusion-derived supervision provides better localization than either single stream, whereas expansion may produce coarse, over-complete proposals.
- Comparison with RefineLoc: TSCN explicitly constrains attention values through normalization loss, unlike RefineLoc.The loss is intended to reduce attention ambiguity.
3 Two-Stream Consensus Network
TSCN combines separate RGB and optical-flow models with iterative pseudo-ground-truth refinement and attention normalization. These components provide frame-level supervision and encourage binary attention for proposal generation.
- Architecture: TSCN extracts RGB and optical-flow snippet features, feeds them into separate base models, and refines both streams iteratively.The two models share an architecture but not parameters.
- Two-Stream Base Models: Attention values estimate each snippet’s likelihood of containing action and weight feature pooling for video-level prediction.The pooled foreground feature is classified by an FC softmax layer.
- Two-Stream Base Models: The attention normalization loss maximizes the gap between top-l and bottom-l attention averages, pushing foreground toward 1 and background toward 0.The overall base-model loss combines classification and attention-normalization terms, weighted by α.
- Pseudo Ground Truth Generation: Late-fusion attention from the two streams generates frame-level pseudo ground truth for the next refinement iteration.The fusion uses a weighted combination of RGB and flow attentions controlled by β.
- Pseudo Ground Truth Generation: Hard pseudo ground truth thresholds fused attention into binary labels, providing stronger supervision while introducing the threshold θ as a hyperparameter.Soft labels retain foreground probabilities but also retain uncertainty; hard labels remove that uncertainty.
- Inference: At test time, TSCN upsamples attention and T-CAM sequences, selects top-k categories, thresholds attention at 0.5, and scores the resulting proposals.The fused video-level prediction and proposal confidence combine outputs from the RGB and flow streams.
4 Experiments
Experiments evaluate TSCN on THUMOS14 and ActivityNet against existing TAL methods, then analyze attention normalization, pseudo ground truth, and qualitative localization behavior.
- Experimental Setup: TSCN is evaluated on THUMOS14 and ActivityNet using UntrimmedNet and I3D features, with fixed pretrained backbones for fair comparison.ActivityNet experiments use available training and validation videos after excluding inaccessible YouTube videos.
- Comparison with the State-of-the-art: On ActivityNet v1.2 and v1.3, TSCN achieves the best average mAP at IoU thresholds 0.5:0.05:0.95 among compared W-TAL methods.The experiments use I3D features and report results on both release versions.
- Ablation Study: The attention normalization loss produces higher attention variance and better localization than background classification loss, while combining both losses lowers localization performance relative to using Latt alone.The ablation attributes the combined-loss degradation to noise from background classification reducing action proposal score accuracy.
- Ablation Study: Hard pseudo ground truth improves localization more than soft pseudo ground truth, and flow supervision outperforms RGB supervision under the same frame-level supervision.The comparison uses different pseudo-ground-truth strategies and refinement iterations on THUMOS14.
- Ablation Study: Hard pseudo ground truth improves localization for both modalities at all IoU thresholds and improves fusion precision, recall, and overall F-measure.The reported changes support its role in reducing false positive action proposals, while the flow stream has a minor recall loss.
- Qualitative Analysis: Qualitative examples show pseudo supervision cleaning fusion attention, separating consecutive action instances, and guiding streams away from false positive temporal locations.The examples cover diving, cricket shot, and soccer penalty videos.
5 Conclusions
TSCN combines iterative refinement with attention normalization to improve weakly supervised temporal action localization. On two benchmarks, it outperforms current state-of-the-art methods, while qualitative results show RGB, flow, and fusion attention sequences alongside action proposals.
- TSCN uses iterative refinement training with frame-level pseudo ground truth as fine-grained supervision for two-stream base models.The pseudo ground truth is generated from late fusion attention and used to iteratively improve the models.
- Figure 4 presents qualitative results using input videos, ground-truth instances, RGB attention, flow attention, fusion attention, and green action-proposal boxes.Time is shown horizontally and attention intensity vertically.
- The attention normalization loss reduces attention ambiguity and leads to more precise action proposals.
- Experiments on THUMOS14 and ActivityNet show that TSCN outperforms current state-of-the-art methods.