Source-linked AI summary
Weakly-supervised Action Localization with Background Modeling
Phuc Xuan Nguyen, Deva Ramanan, Charless C. Fowlkes
TL;DR
Weakly-supervised action localization must identify frame-level actions from videos labeled only at the whole-video level. The paper models both foreground and background, uses class-specific activation maps to supervise class-agnostic attention, and supplements training with microvideos. It reports improved localization over prior systems and shows that added microvideos improve performance, while precise boundaries remain difficult.
Problem
The paper studies how to localize actions in long sequences when training videos provide only video-level labels rather than exact temporal boundaries.
Method
The approach explicitly models foreground and background frames, uses top-down class-specific activation maps to guide bottom-up attention, and incorporates microvideos as supplemental weakly-supervised data.
Results
The model improves mAP by 10% over STPN for IoUs below 0.5 and is competitive with fully-supervised systems in the lower-IoU regime.
Takeaways & Limitations
Microvideos can scale weakly-supervised training and improve action localization, particularly by improving recognition of action instances.
Takeaways & Limitations
Without boundary annotations, the model identifies where actions occur but does not articulate precise temporal boundaries as well as fully-supervised methods.
Abstract
from arXiv · showhide
We describe a latent approach that learns to detect actions in long sequences given training videos with only whole-video class labels. Our approach makes use of two innovations to attention-modeling in weakly-supervised learning. First, and most notably, our framework uses an attention model to extract both foreground and background frames whose appearance is explicitly modeled. Most prior works ignore the background, but we show that modeling it allows our system to learn a richer notion of actions and their temporal extents. Second, we combine bottom-up, class-agnostic attention modules with top-down, class-specific activation maps, using the latter as form of self-supervision for the former. Doing so allows our model to learn a more accurate model of attention without explicit temporal supervision. These modifications lead to 10% AP@IoU=0.5 improvement over existing systems on THUMOS14. Our proposed weaklysupervised system outperforms recent state-of-the-arts by at least 4.3% AP@IoU=0.5. Finally, we demonstrate that weakly-supervised learning can be used to aggressively scale-up learning to in-the-wild, uncurated Instagram videos. The addition of these videos significantly improves localization performance of our weakly-supervised model
1. Introduction
The paper addresses weakly-supervised action localization from video-level labels by modeling both foreground and background frames, guiding bottom-up attention with class-specific cues, and using microvideos as supplemental training data.
- Problem: Weakly-supervised action localization learns action locations in long sequences using only video-level class labels.The task avoids requiring exact temporal boundaries during training.
- Background modeling: Explicitly modeling background frames improves weakly-supervised localization beyond foreground-only attention.The approach extends attention-based systems by representing frames that are not selected as action foreground.
- Top-down guided attention: Top-down class-specific activation maps supervise class-agnostic bottom-up attention to refine temporal attention estimates.The loss encourages the two attention signals to agree for classes present in each training video.
- Micro-videos as training supplements: Supplemental microvideos provide weakly-supervised training data that improves action localization while enabling aggressive scaling.The paper treats user-generated tags as loosely corresponding to video-level labels.
- Contributions: The system extends prior weakly-supervised localization models with background modeling and top-down class-guided attention.These are identified as core contributions alongside comparative evaluation and supplemental-data experiments.
- Evaluation: Comparative analyses evaluate weakly- and fully-supervised localization systems on THUMOS14 and ActivityNet.The paper reports comparisons across both supervision settings and datasets.
2. Related Works
Related work contrasts expensive boundary-annotated localization with weakly-supervised methods that learn from video-level labels and attention mechanisms.
- Fully-supervised localization: Fully-supervised localization commonly uses annotated boundaries in two-stage propose-then-classification frameworks.These methods rely on human-provided temporal annotations within longer videos.
- Background modeling: Background frames can serve as negative examples for building a background model in localization systems.This contrasts with approaches that focus primarily on action regions.
- Weak supervision: Weakly-supervised methods replace costly temporal boundaries with video-level labels and use selection or attention modules to identify important segments.Examples include UntrimmedNets, Hiden-Seek, and STPN.
3. Localization from Weak Supervision
The model learns frame-level action localization from video-level labels by combining attention-weighted foreground pooling with explicit background modeling, class-guided attention, and foreground-background separation. At inference, attention-derived proposals are scored with class activation maps and filtered with class-wise suppression.
- Architecture: The model maps short-segment features to scalar attention weights that pool foreground frames into a video-level representation for classification.The attention function uses fully connected layers with ReLU and sigmoid activations.
- Background modeling: The complement of attention pools presumed background frames, which the same classifier is trained to assign to the background class.This background-aware loss helps the classifier distinguish actions from background rather than learning only foreground classification.
- Self-guided attention: Class-specific T-CAMs provide self-supervision that guides the class-agnostic attention map toward action-relevant temporal regions.The top-down targets are smoothed temporally, and multiple actions can be combined by max-pooling their attention targets.
- Inference: Detection thresholds video-level class probabilities, forms connected proposals from attention thresholds, scores proposals with weighted T-CAMs, and applies per-class non-maxima suppression.Using multiple attention thresholds and averaged multimodal attention weights provides a larger proposal pool.
- Foreground-background separation: A bottom-up clustering loss encourages foreground and background pooled features to remain distinct without using video-level labels.The loss uses separate foreground and background parameters and encourages both classifiers to respond to their corresponding pooled features.
- Ablation: Ablations show that each additional loss improves localization, while combining the losses yields further gains because their cues are complementary.The ablation table includes baseline rows obtained from STPN.
4. Experiments
The experiments evaluate weakly supervised action localization on THUMOS14 and ActivityNet1.3, including supplemental Instagram microvideos, under standard mAP-at-IoU evaluation.
- Datasets: The method is evaluated on THUMOS14 and ActivityNet1.3 action localization benchmarks.THUMOS14 training uses validation videos without temporal annotations, while ActivityNet1.3 provides 200 activity classes and uses preextracted I3D features matching STPN.
- Supplemental data: Instagram microvideos provide loosely labeled, short, untrimmed clips for supplemental weakly supervised training.The curated set contains 915 videos lasting 6–15 seconds, and these clips are added to the THUMOS14 training set without changing the rest of the experiment.
- Evaluation: Performance is measured with mean Average Precision at multiple temporal intersection-over-union thresholds.Evaluation follows the standard temporal action localization protocol and uses ActivityNet benchmarking code.
- Implementation: The implementation follows STPN-like settings with two-stream I3D segment features and dense optical flow preprocessing.The RGB and optical-flow streams use I3D features from Kinetics-trained networks, with Farneback optical flow for the flow stream.
- Inference: Testing rejects classes below video-level probability 0.1 and combines proposals generated across thresholds from 0 to 0.5 using NMS at 0.5.If no foreground class exceeds 0.1, proposals use the highest-probability foreground class.
5. Results
The results show that background modeling, self-guided attention, and combined loss terms improve weakly supervised localization across benchmarks, while precise boundaries remain difficult without temporal annotations.
- Ablation: Combining the loss terms produces better localization than individual losses, indicating complementary cues.The ablation study reports that adding each loss improves localization performance and that their combination performs better.
- Qualitative comparison: The model detects instances missed by STPN and produces more accurate temporal boundaries for instances detected by both methods.Improved detections increase recall and average precision, while more accurate boundaries improve AP at stricter IoU thresholds.
- THUMOS14: 10% mAP improvement over STPN is reported on THUMOS14 for IoUs below 0.5.The method also significantly outperforms other weakly supervised approaches and is comparable to fully supervised systems in lower-IoU regimes.
- ActivityNet1.3: On ActivityNet1.3, the method significantly outperforms existing weakly supervised approaches while remaining competitive with fully supervised methods.The comparison uses the validation set results reported in Table 3.
- Microvideos: At least 3% AP@IoU=0.5 improvement occurs for five low-frequency action categories after adding microvideos.Microvideos rebalance the skewed instance distribution and improve recognition, but do not improve highly precise boundaries at higher IoU thresholds.
- Failure modes: Weak supervision struggles to separate adjacent actions, decompose composite actions, and match subjective human boundary annotations.These failure modes constrain performance when little background separates actions and at higher IoU regimes.
- Attention learning: Top-down T-CAM self-supervision lets the model train to convergence instead of relying on the sparsity-driven behavior of STPN.The top-down signals supervise bottom-up attention weights, providing an alternative to selecting only the smallest number of frames needed for video labels.
- Scaling weak supervision: The approach uses additional weakly supervised online data despite domain shift between Instagram videos and THUMOS14.The conclusion reports improved performance across many categories and connects this scaling ability to reduced reliance on expensive video annotation.