Source-linked AI summary
Weakly-Supervised Action Localization by Generative Attention Modeling
Baifeng Shi, Qi Dai, Yadong Mu, Jingdong Wang
TL;DR
Weakly supervised action localization must distinguish actions from nearby context using only video-level labels, but classification-based attention often confuses the two. DGAM combines discriminative attention with a conditional VAE that models frame representations conditioned on attention; experiments on THUMOS14 and ActivityNet1.2 report new state-of-the-art results on both datasets.
Problem
Video-level supervision leaves WSAL classifiers vulnerable to action-context confusion because nearby context frames can resemble the target action.
Method
DGAM combines discriminative attention modeling with a conditional VAE that models class-agnostic frame-wise representation distributions conditioned on attention.
Results
DGAM achieves new state-of-the-art results on both THUMOS14 and ActivityNet1.2.
Takeaways & Limitations
Modeling representation distributions conditioned on attention separates action and context frames in weakly-supervised localization.
Takeaways & Limitations
At low IoU thresholds, DGAM may discard out-of-distribution hard candidates, allowing another model to perform better at IoU=0.1 and 0.2.
Abstract
from arXiv · showhide
Weakly-supervised temporal action localization is a problem of learning an action localization model with only video-level action labeling available. The general framework largely relies on the classification activation, which employs an attention model to identify the action-related frames and then categorizes them into different classes. Such method results in the action-context confusion issue: context frames near action clips tend to be recognized as action frames themselves, since they are closely related to the specific classes. To solve the problem, in this paper we propose to model the class-agnostic frame-wise probability conditioned on the frame attention using conditional Variational Auto-Encoder (VAE). With the observation that the context exhibits notable difference from the action at representation level, a probabilistic model, i.e., conditional VAE, is learned to model the likelihood of each frame given the attention. By maximizing the conditional probability with respect to the attention, the action and non-action frames are well separated. Experiments on THUMOS14 and ActivityNet1.2 demonstrate advantage of our method and effectiveness in handling action-context confusion problem. Code is now available on GitHub.
1. Introduction
Weakly supervised action localization avoids costly frame-level annotation but suffers from action-context confusion. DGAM addresses this by combining discriminative attention with conditional generative modeling of frame representations.
- Weakly-supervised action localization uses only video-level category labels, avoiding expensive and time-consuming frame-level interval annotation.
- Existing top-down and bottom-up approaches largely rely on video-level classification to infer frame attention.
- Context frames near actions can resemble the target category, causing classifiers to select non-action frames without frame-wise annotations.
- Action and context differ in appearance, motion, and representation features, regardless of the action category.
- DGAM combines discriminative attention modeling with a conditional VAE that models class-agnostic frame distributions conditioned on attention.
- By maximizing representation likelihood, DGAM refines frame-wise attention to separate action from context.
2. Related Works
Related work covers conventional video recognition, fully supervised localization, weakly supervised pipelines, and generative modeling. Existing WSAL methods leave action-context confusion insufficiently addressed, while prior context separation relies on a strong stationary-context assumption.
- Video action recognition commonly uses learned feature extractors, including two-stream networks, TSN, C3D, P3D, and I3D.
- Fully-supervised localization methods mainly use two-stage proposal classification or one-stage direct prediction of action categories and locations.
- WSAL methods are grouped into top-down pipelines based on classification activation and bottom-up methods that directly predict temporal attention.
- Existing WSAL approaches generally ignore action-context confusion caused by the absence of frame-wise labels.
- Prior hard-negative mining assumes context clips are stationary, an assumption described as limited and potentially harmful to prediction.
- VAEs model data distributions by optimizing a variational lower bound on marginal likelihood.
3. Method
DGAM learns frame attention through both video-level discrimination and generative modeling of frame representations conditioned on attention. It alternates between optimizing attention-related modules and a CVAE, using predicted attention as pseudo-labels because frame-level ground truth is unavailable.
- Attention-based Framework: The method learns attention λ directly from sampled frame features and video-level labels, with λ_t ∈ [0, 1] representing each frame’s attention.The formulation assumes one video-level class for brevity, while allowing extension to multi-label videos.
- Attention-based Framework: DGAM optimizes attention using both classification likelihood and frame-representation likelihood, encouraging different attentions for frames with different representations.The discriminative term favors action-class prediction, while the generative term reconstructs frame features from attention.
- Discriminative Attention Modeling: The discriminative module pools frame features with attention for foreground and with 1−λ for background, then jointly trains attention and classification modules.Its loss rewards discriminative foreground features and penalizes discriminative background features.
- Optimization: DGAM alternates between updating attention and classification modules with the CVAE fixed, and training the CVAE with attention predictions as pseudo-labels.The alternating procedure addresses the absence of ground-truth frame attention and repeats both stages for several iterations.
- Generative Attention Modeling: The generative module uses a CVAE to model each frame representation from attention λ_t and latent variable z_t, with Gaussian conditional and attention-dependent prior distributions.The encoder approximates qφ(z_t|x_t, λ_t), while the decoder models pψ(x_t|λ_t, z_t).
- Action Prediction: At inference, high-attention frames are grouped into consecutive segments, whose class scores are refined by subtracting scores from surrounding segments.The final segment score s*(t_s, t_e, c) uses subtraction parameter η.
4. Experiments
Experiments on THUMOS14 and ActivityNet1.2 evaluate DGAM under weak supervision, showing that generative attention modeling improves localization, especially by reducing action-context confusion.
- Experimental Setup: Experiments use THUMOS14 and ActivityNet1.2 with only video-level category labels during training, reporting mAP at multiple IoU thresholds.THUMOS14 results are averaged over five runs, and ActivityNet1.2 uses its standard validation protocol.
- Attention Evaluation: The new attention module substantially improves THUMOS14 localization, even when paired with the old classifier.The new classification module provides little or no improvement, indicating that DGAM’s main benefit comes from learned attention values.
- Attention Evaluation: GAM reduces both falsely captured and omitted frames while better filtering classifier false positives such as context frames.It contributes little to recovering action frames omitted by the classifier, consistent with its design emphasis on excluding false positives.
- Ablation Studies: Adding background classification, self-guided regularization, and generative attention yields gains of 3.3%, 1.9%, and 2.1% mAP, reaching 28.8% on THUMOS14.The generative contribution is introduced by adding the reconstruction loss and optimizing the conditional VAE objective.
- Parameter Evaluation: DGAM performance is robust to the tested attention-discrepancy and CVAE β settings, while latent dimension peaks at 2^7 = 128 before declining.The decline is partially attributed to sparsity in the limited data available for high-dimensional latent spaces.
- State-of-the-Art Comparisons: DGAM improves over state-of-the-art weakly supervised methods by 2% at IoU=0.5 on THUMOS14 and by 2% on ActivityNet1.2 mAP@AVG.It performs especially well at high IoU thresholds and is comparable to several fully supervised methods.
5. Conclusion
The paper presents DGAM, using conditional feature-distribution modeling to address action-context confusion in weakly-supervised action localization. Experiments on THUMOS14 and ActivityNet1.2 validate the method and report new state-of-the-art results on both datasets.
- 5. Conclusion: DGAM addresses action-context confusion in weakly-supervised action localization by modeling frame-wise attention from frame-feature distributions.The method uses a conditional variational auto-encoder to construct feature distributions conditioned on different attentions and refine attention according to frame features.
- 5. Conclusion: The method is motivated by the observation that context and action features differ clearly at the representation level.This distinction supports separating action and non-action frames through conditional feature-distribution modeling.
- 5. Conclusion: Experiments on THUMOS14 and ActivityNet1.2 validate the proposed method and analysis.
- 5. Conclusion: DGAM achieves new state-of-the-art results on both benchmark datasets.