Source-linked AI summary
Background Suppression Network for Weakly-supervised Temporal Action Localization
Pilhyeon Lee, Youngjung Uh, Hyeran Byun
TL;DR
Weakly-supervised temporal action localization lacks frame-wise labels, and treating background as an action class can produce false positives. BaS-Net adds a background class and trains asymmetrical, weight-sharing Base and Suppression branches to filter background activations, outperforming state-of-the-art methods on THUMOS’14 and ActivityNet.
Problem
Weakly-supervised temporal action localization must infer frame-wise labels from video-level labels, but background frames are pushed toward action classes when background is not separately modeled.
Method
BaS-Net introduces an auxiliary background class and an asymmetrical two-branch weight-sharing architecture with filtering and contrasting objectives to suppress background activations.
Results
BaS-Net outperforms current state-of-the-art weakly-supervised temporal action localization methods on THUMOS’14 and ActivityNet.
Takeaways & Limitations
Background modeling and the contrasting two-branch training strategy are both necessary for performance improvement and background suppression.
Takeaways & Limitations
STAR cannot be directly compared with BaS-Net because it uses additional annotations for the frequency of action instances.
Abstract
from arXiv · showhide
Weakly-supervised temporal action localization is a very challenging problem because frame-wise labels are not given in the training stage while the only hint is video-level labels: whether each video contains action frames of interest. Previous methods aggregate frame-level class scores to produce video-level prediction and learn from video-level action labels. This formulation does not fully model the problem in that background frames are forced to be misclassified as action classes to predict video-level labels accurately. In this paper, we design Background Suppression Network (BaS-Net) which introduces an auxiliary class for background and has a two-branch weight-sharing architecture with an asymmetrical training strategy. This enables BaS-Net to suppress activations from background frames to improve localization performance. Extensive experiments demonstrate the effectiveness of BaS-Net and its superiority over the state-of-the-art methods on the most popular benchmarks - THUMOS'14 and ActivityNet. Our code and the trained model are available at https://github.com/Pilhyeon/BaSNet-pytorch.
1 Introduction
Weakly-supervised temporal action localization avoids costly frame-level annotation but struggles because background frames are not modeled separately. BaS-Net addresses this with an auxiliary background class and an asymmetrical, weight-sharing two-branch design that suppresses background activations.
- Full supervision for temporal action localization is expensive, subjective at action boundaries, and error-prone, motivating weakly-supervised alternatives.
- Video-level supervision treats each video as positive for its action classes, and multiple-instance learning derives frame-wise scores from these labels.
- Without a separate background class, background frames are pushed toward video action classes, causing false positives and performance degradation.
- BaS-Net introduces an auxiliary background class so frames can target categories beyond the video's action labels.
- Its Base and Suppression branches share weights, while a filtering module attenuates background features and contrasting objectives minimize background scores in the Suppression branch.
- BaS-Net outperforms current state-of-the-art weakly-supervised methods on THUMOS’14 and ActivityNet, while ablations find both background modeling and joint contrasting training necessary.
2 Related Work
Temporal action localization has been studied under full and weak supervision. Weakly-supervised methods use video-level labels to derive frame-wise class activation sequences, while the proposed method overview includes feature extraction, a base branch, and a suppression branch.
- Fully-supervised temporal action localization (TAL): Fully-supervised temporal action localization requires both action classes and their temporal intervals.
- Fully-supervised temporal action localization (TAL): Previous fully-supervised methods commonly generate proposals with sliding windows and classify them into C + 1 classes, including background.
- Weakly-supervised temporal action localization (WTAL): Weakly-supervised temporal action localization addresses the same task with less supervision, such as video-level labels.
- Weakly-supervised temporal action localization (WTAL): Weakly-supervised methods generate class activation sequences to derive frame-wise scores from video-level labels.
- Overview of the proposed method: The proposed method consists of feature extraction, a base branch, and a suppression branch.
3 Proposed Method
BaS-Net models background explicitly through a two-branch, weight-sharing architecture with asymmetric objectives, then uses the suppression branch to localize actions.
- Background class: BaS-Net adds an auxiliary background class because every untrimmed training video contains background frames, creating a background-positive label for each video.The video-level label therefore has C + 1 dimensions, with the additional background entry set to positive.
- Joint training: Weight sharing couples the branches while their objectives differ: the Base branch predicts background positively, whereas the Suppression branch minimizes its background score while preserving action classification.This makes the filtering module the mechanism for reconciling the contrasting objectives and suppressing background activations.
- Base branch: The Base branch predicts class activation sequences (CAS) for action and background classes from frame-wise features, then aggregates segment scores with top-k mean.Video-level class scores are converted into class probabilities with softmax and trained using binary cross-entropy.
- Suppression branch: The Suppression branch applies a temporal filtering module that produces foreground weights and attenuates background features before the shared convolutional architecture.The filtered feature map is formed by element-wise multiplication over the temporal dimension, after which the remaining process parallels the Base branch.
- Joint training: The branches are jointly trained with a weighted overall loss, including base loss, suppression loss, and L1 normalization of attention weights.The normalization is used to make foreground weights more polarized.
- Inference: At test time, BaS-Net uses the Suppression branch, filters predicted classes and CAS by separate thresholds, and converts consecutive candidate segments into proposals.Proposal confidence is computed from contrast between inner and outer areas.
4 Experiments
Experiments evaluate BaS-Net on THUMOS’14 and ActivityNet using video-level labels for training and temporal annotations only for evaluation. BaS-Net consistently outperforms weakly-supervised methods, while ablations and qualitative results support the joint background-suppression design.
- Experimental settings: Experiments use THUMOS’14 and ActivityNet, training only with video-level labels and reserving temporal annotations for evaluation.Evaluation uses mean average precision at multiple IoU thresholds.
- Comparison with state-of-the-art methods: BaS-Net significantly outperforms same-supervision methods on THUMOS’14 with both UntrimmedNet and I3D features.It also shows the smallest gap to the latest fully supervised methods and exceeds several at some IoU thresholds.
- Comparison with state-of-the-art methods: On ActivityNet1.3, BaS-Net outperforms all other weakly-supervised approaches and surpasses STAR at every IoU threshold despite weaker labels.STAR uses additional action-instance-frequency annotations.
- Comparison with state-of-the-art methods: On ActivityNet1.2, BaS-Net outperforms all weakly-supervised methods and follows the fully supervised method with a small gap.The comparison reports average mAP across IoU thresholds 0.5:0.05:0.95.
- Ablation study: Ablations show that adding only the background class decreases performance, whereas the Suppression branch improves localization and the complete BaS-Net performs best.Joint training of both branches with contrasting objectives is required to learn background and action classes effectively.
- Qualitative results: Qualitative examples show background suppression improving localization for sparse actions, frequent actions, and backgrounds resembling foreground.The Suppression branch attenuates background activations where the Base branch can assign them higher scores than foreground.
5 Conclusion
The paper addresses missing background modeling in weakly-supervised temporal action localization by adding a background class and a two-branch asymmetrical training strategy. Experiments show the framework suppresses background and outperforms state-of-the-art methods on THUMOS’14 and ActivityNet.
- BaS-Net classifies both action classes and a background class within multiple instance learning.This explicitly models background frames during weakly-supervised localization.
- The method introduces a two-branch architecture with an asymmetrical training strategy to exploit background information.Both components are presented as part of the proposed solution.
- Ablation studies show that the background class and training strategy are both necessary for performance improvement.
- BaS-Net effectively suppresses background and outperforms current state-of-the-art methods on THUMOS’14 and ActivityNet.