Source-linked AI summary
3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization
Sanath Narayan, Hisham Cholakkal, Fahad Shahbaz Khan, Ling Shao
TL;DR
Temporal action localization usually relies on costly frame-level supervision, motivating weakly supervised methods based on video-level labels and counts. 3C-Net jointly uses classification, adapted multi-label center, and counting losses to improve localization. Across THUMOS14 and ActivityNet 1.2, it sets a new state-of-the-art, including a 4.6% absolute mAP gain on THUMOS14.
Problem
Temporal action localization commonly requires costly frame-level boundary annotations, whereas video-level action labels are cheaper and less subjective.
Method
3C-Net jointly optimizes classification, adapted multi-label center, and counting losses using video-level action labels and counts.
Results
3C-Net sets a new state-of-the-art for weakly-supervised temporal action localization on THUMOS14 and ActivityNet 1.2, with a 4.6% absolute mAP gain on THUMOS14.
Takeaways & Limitations
The joint formulation improves weakly-supervised action localization and classification across the two evaluated benchmarks.
Abstract
from arXiv · showhide
Temporal action localization is a challenging computer vision problem with numerous real-world applications. Most existing methods require laborious frame-level supervision to train action localization models. In this work, we propose a framework, called 3C-Net, which only requires video-level supervision (weak supervision) in the form of action category labels and the corresponding count. We introduce a novel formulation to learn discriminative action features with enhanced localization capabilities. Our joint formulation has three terms: a classification term to ensure the separability of learned action features, an adapted multi-label center loss term to enhance the action feature discriminability and a counting loss term to delineate adjacent action sequences, leading to improved localization. Comprehensive experiments are performed on two challenging benchmarks: THUMOS14 and ActivityNet 1.2. Our approach sets a new state-of-the-art for weakly-supervised temporal action localization on both datasets. On the THUMOS14 dataset, the proposed method achieves an absolute gain of 4.6% in terms of mean average precision (mAP), compared to the state-of-the-art. Source code is available at https://github.com/naraysa/3c-net.
1. Introduction
3C-Net targets weakly-supervised temporal action localization using video-level action labels and counts. It combines classification, multi-label center, and counting losses to improve feature discriminability and separate adjacent action instances.
- Motivation: Temporal action localization requires action boundaries, but frame-level annotations are costly and subjective compared with video-level labels.The paper therefore focuses on localization using weak supervision.
- Proposed framework: The adapted center loss uses class-specific attention-based feature aggregation to apply center loss to multi-label action videos.It reduces intra-class feature variation by penalizing distances from features to class-specific centers.
- Qualitative evidence: In a PoleVault example, classification alone merges four adjacent actions, center loss partially separates them, and 3C-Net delineates all four instances.The figure compares CLS, CLS + CL, and CLS + CL + CT against ground-truth segments.
- Proposed framework: The counting loss uses video-level action counts to encourage positive scores within action instances and zero scores elsewhere.This improves separation of adjacent instances without requiring temporal boundary annotations.
- Proposed framework: 3C-Net jointly optimizes classification, multi-label center, and counting losses for weakly-supervised action localization.Classification promotes inter-class separability, center loss improves feature discriminability, and counting loss separates adjacent instances.
- Empirical results: Experiments on THUMOS14 and ActivityNet 1.2 show improved localization over the classification-only baseline and a new state-of-the-art on both datasets.The paper reports a 4.6% absolute mAP gain over the best existing weakly-supervised method on THUMOS14.
2. Related Work
Related work uses varied weak supervision for temporal action localization, including action categories, scripts, subtitles, sparse points, and action order. Existing methods also commonly combine appearance and motion features with classification and temporal-selection components.
- Weak supervision: Weak supervision for action localization has included video action categories, movie scripts, sparse spatio-temporal points, subtitles, actor-action pairs, and action order.These approaches seek temporal or spatio-temporal localization without full frame-level boundary supervision.
- Category-label supervision: Several methods use video-level category labels, including approaches based on classification, temporal selection, or co-activity similarity.The cited methods distinguish relevant temporal segments using different feature-learning and selection formulations.
- Feature representations: State-of-the-art weakly-supervised methods typically use appearance and motion features extracted from action-recognition backbones.Examples include two-stream architectures and temporal class activation maps.
3. Method
3C-Net combines RGB and flow streams with classification, multi-label center, and counting losses. These components learn discriminative temporal features, fuse stream-specific T-CAMs, and improve separation of adjacent action instances.
- Overall Architecture: 3C-Net extracts appearance and motion features in parallel RGB and flow streams, each using fully connected layers to produce temporal class activation maps.The streams are fused later using learned class-specific weights.
- Classification Loss: The classification loss promotes inter-class separability from temporal class activation maps using top-k temporal pooling and category-wise probability distributions.The pooling ratio uses k = ⌈s_i/8⌉ segments, proportional to video length.
- Center Loss for Multi-label Classification: The multi-label center loss clusters category-specific features around learned class centers after attention-based aggregation of high-response temporal regions.Thresholding attention by a category-specific median selects regions for aggregation, while absent categories are not aggregated.
- Counting Loss: The counting loss uses auxiliary action-instance counts to improve the delineation of temporally adjacent action instances through the fused final T-CAM.It compares predicted and ground-truth counts, with λ typically set to 10^-3 to balance positive and negative instances.
- Overall Objective: The three loss terms are jointly optimized end-to-end to enhance feature separability, feature discriminability, and adjacent-instance delineation.The resulting representation is a more discriminative temporal class activation map.
4. Experiments
Experiments on THUMOS14 and ActivityNet 1.2 evaluate 3C-Net against existing methods, baselines, ablations, and qualitative examples. The results show improved weakly supervised localization, with gains from the center and counting losses and strong performance across varied action scenarios.
- State-of-the-art comparison: 3C-Net achieves a 4.6% absolute mAP gain over the best weakly-supervised THUMOS14 result at IoU=0.5.The method also improves consistently at other IoU thresholds.
- Baseline Comparison and Ablation Study: 7.5% absolute mAP improvement over the classification-only baseline is obtained on THUMOS14 at IoU=0.5.The baseline scores 19.1 mAP, while the full formulation reaches 26.6 mAP.
- Baseline Comparison and Ablation Study: Removing classification, center, or negative-category counting losses reduces mAP by 2.5%, 2.1% or 1.5%, respectively.Using absolute rather than relative counting error also reduces performance by 1.2%.
- Qualitative Analysis: Qualitative examples show localization of repeated actions, visually similar overlapping categories, and long-duration activities using T-CAM detections.Examples include HighJump, Shotput, ThrowDiscus, Playing Violin, and Parallel Bars.
5. Conclusion
3C-Net combines classification, multi-label center, and counting losses for weakly-supervised action localization. Experiments on two challenging datasets support its effectiveness for action localization and classification.
- 3C-Net combines classification, center, and counting loss terms for weakly-supervised action localization.The formulation jointly addresses feature separability, discriminability, and action-instance separation.
- A class-specific attention-based feature aggregation strategy enables multi-label videos to train the center loss.
- The counting loss leverages video-level action-count information.
- Experiments on two challenging datasets demonstrate the approach’s effectiveness for action localization and classification.