Source-linked AI summary
CoLA: Weakly-Supervised Temporal Action Localization with Snippet Contrastive Learning
Can Zhang, Meng Cao, Dongming Yang, Jie Chen, Yuexian Zou
TL;DR
WS-TAL models can misclassify ambiguous boundary snippets when processing snippets independently, disrupting action intervals. CoLA mines potential hard snippets near boundaries and applies SniCo contrastive learning with easy snippets to refine their representations; experiments validate state-of-the-art performance on THUMOS’14 and ActivityNet v1.2.
Problem
Independent snippet processing in WS-TAL leaves ambiguous snippets vulnerable to misclassification, causing broken or inaccurately extended action intervals.
Method
CoLA uses boundary-aware Hard Snippet Mining to locate potential hard snippets and SniCo Loss to refine them by comparing with easy snippets.
Results
Experiments on THUMOS’14 and ActivityNet v1.2 validate CoLA’s state-of-the-art performance.
Takeaways & Limitations
SniCo improves feature representation for WS-TAL, with its introduction producing a 7.5% gain in mAP@0.5 over action-loss-only supervision.
Takeaways & Limitations
The reported experimental results were later improved through fine-tuning some hyper-parameter settings, with details deferred to the code.
Abstract
from arXiv · showhide
Weakly-supervised temporal action localization (WS-TAL) aims to localize actions in untrimmed videos with only video-level labels. Most existing models follow the "localization by classification" procedure: locate temporal regions contributing most to the video-level classification. Generally, they process each snippet (or frame) individually and thus overlook the fruitful temporal context relation. Here arises the single snippet cheating issue: "hard" snippets are too vague to be classified. In this paper, we argue that learning by comparing helps identify these hard snippets and we propose to utilize snippet Contrastive learning to Localize Actions, CoLA for short. Specifically, we propose a Snippet Contrast (SniCo) Loss to refine the hard snippet representation in feature space, which guides the network to perceive precise temporal boundaries and avoid the temporal interval interruption. Besides, since it is infeasible to access frame-level annotations, we introduce a Hard Snippet Mining algorithm to locate the potential hard snippets. Substantial analyses verify that this mining strategy efficaciously captures the hard snippets and SniCo Loss leads to more informative feature representation. Extensive experiments show that CoLA achieves state-of-the-art results on THUMOS'14 and ActivityNet v1.2 datasets. CoLA code is publicly available at https://github.com/zhang-can/CoLA.
1. Introduction
WS-TAL methods commonly classify fixed-size snippets independently and threshold their temporal activations, but ambiguous snippets can disrupt or extend localized action intervals. CoLA addresses this issue by mining boundary-adjacent hard snippets and refining them through contrastive comparison with easy snippets.
- WS-TAL uses video-level labels to localize and classify action intervals in untrimmed videos, reducing the manual labeling burden of fully supervised TAL.
- Most methods classify fixed-size snippets over time, then threshold and merge class activations to produce localization results.
- Ambiguous snippets can be misclassified individually, breaking action intervals or extending their boundaries inaccurately.
- Comparing hard snippets with easy action or background snippets helps distinguish their categories and correct their inferred labels.
- CoLA mines potential hard snippets near action boundaries using thresholding, temporal dilation, and erosion before applying contrastive feature refinement.
- The paper introduces SniCo Loss and Hard Snippet Mining for weakly supervised localization, reporting effectiveness on THUMOS’14 and ActivityNet v1.2.
2. Related Work
Related work covers fully supervised and weakly supervised localization, background and context modeling, and contrastive representation learning. CoLA applies contrastive learning to WS-TAL to refine hard snippet representations.
- Fully supervised localization uses frame-level annotations, with proposal-based and frame-based approaches differing in how they generate and classify temporal predictions.
- Weakly supervised methods use video-level annotations and include clip proposal classification, sparsity constraints, and strategies that expand or suppress discriminative regions.
- Prior work addresses action-context confusion through hard negatives, multi-branch classification, attention, background modeling, or auxiliary background classes.
- Contrastive representation learning brings associated signals together and separates unassociated ones in an embedding space using noise contrastive estimation.
- CoLA’s pipeline combines feature extraction, actionness modeling, hard and easy snippet mining, and training with Action and SniCo losses.
- CoLA is presented as the first application of noise contrastive estimation to WS-TAL, with experiments reporting refined hard snippet representations that benefit localization.
3. Method
CoLA follows a pipeline that extracts and embeds snippet features, models actionness, mines hard and easy snippets, and trains with complementary losses before inference.
- CoLA consists of feature extraction, actionness modeling, hard and easy snippet mining, network optimization, and a subsequent inference process.The optimization uses Action Loss and Snippet Contrast Loss.
3.1. Feature Extraction and Embedding
The method represents each untrimmed video as sampled, non-overlapping multiframe snippets and embeds extracted RGB features for downstream temporal modeling.
- Each untrimmed video has a multi-hot video-level label vector and is divided into multiframe, non-overlapping snippets.
- RGB features are extracted from the sampled snippets using a pretrained feature extractor such as I3D.
- The extracted snippet features are concatenated and passed through an embedding function to obtain embedded features.
- The embedding function is implemented with a temporal convolution followed by a ReLU activation.
3.2. Actionness Modeling
CoLA models snippet-level actionness by aggregating class-specific temporal activations into a class-agnostic score for each snippet.
- The model applies a temporal convolutional classifier to embedded features, producing T-CAS values for each temporal snippet and action category.
- It sums T-CAS across classes and applies a Sigmoid function to obtain class-agnostic actionness scores.
- This actionness representation avoids an additional binary classifier because T-CAS already contains snippet-level class-specific predictions.
3.3. Hard & Easy Snippet Mining
CoLA mines hard snippets near action boundaries and easy snippets from extreme actionness scores, enabling contrastive comparisons under weak supervision.
- Hard Snippet Mining: Boundary-adjacent snippets are treated as potentially hard because transitional areas between action and background produce ambiguous detections.
- Hard Snippet Mining: Mined hard snippets are divided into hard action and hard background sets according to their locations.
- Hard Snippet Mining: The method thresholds actionness into a binary action-background sequence before mining boundary regions.
- Hard Snippet Mining: It uses cascaded erosion and dilation operations, with differential inner and outer regions defining hard action and hard background areas.
- Easy Snippet Mining: Easy action and background snippets are selected from the top-k and bottom-k actionness scores, excluding hard snippet regions.
3.4. Network Training
CoLA trains with the standard Action Loss plus SniCo Loss, which contrasts hard snippets with easy snippets from the same category to refine their representations.
- CoLA adds SniCo Loss to the baseline Action Loss using mined hard and easy snippets, improving performance over the baseline model.
- The total training loss is Ltotal = La + λLs, where La is Action Loss, Ls is SniCo Loss, and λ balances them.
- Action Loss: Action Loss is computed from video-level class predictions obtained by averaging the top-k class-specific T-CAS scores.
- Snippet Contrast (SniCo) Loss: SniCo Loss forms hard-action and hard-background contrastive refinements using corresponding easy snippets and sampled negatives.
- Snippet Contrast (SniCo) Loss: The contrastive objective maximizes mutual information between easy and hard snippets of the same category, refining features and alleviating single snippet cheating.
3.5. Inference
During inference, CoLA converts snippet activations into localized proposals by selecting confident categories, thresholding their T-CAS, grouping continuous snippets, and applying NMS.
- The method aggregates top-keasy snippet scores to obtain video-level predictions and retains categories whose scores exceed θv.
- For each selected category, it thresholds the corresponding T-CAS with θs, groups continuous snippets into proposals, and applies NMS to remove duplicates.
4. Experiments
Experiments on THUMOS’14 and ActivityNet v1.2 show that CoLA improves weakly-supervised temporal action localization, while ablations support its SniCo Loss and Hard Snippet Mining components.
- Comparison with State-of-the-Arts: CoLA achieves 32.2% mAP@0.5 and 40.9% mAP@AVG on THUMOS’14, outperforming previous weakly-supervised methods at all IoU thresholds.
- Comparison with State-of-the-Arts: CoLA shows significant improvements over state-of-the-art weakly-supervised methods on ActivityNet v1.2 while remaining competitive with fully-supervised methods.
- Ablation Studies: Introducing SniCo Loss increases mAP@0.5 by 7.5% relative to the baseline using only action loss supervision.UMAP visualizations show more separable action and background embeddings, especially for ambiguous hard snippets.
- Ablation Studies: Removing either HA or HB refinement causes a dramatic performance drop, indicating that both refinements contribute to improved performance.
- Ablation Studies: Mean RDO drops across all tested δ scales from epoch 0 to epoch 2k, reaching 3.7% at δ = 0.2.This indicates that mined hard snippets are captured more precisely as training proceeds and mostly lie in error-prone areas.
- Ablation Studies: Performance improves as the negative sample size S increases, while mask-size results remain stable across a wide range and peak at M = 6 and m = 3.
5. Conclusion
CoLA addresses single snippet cheating in weakly-supervised action localization by mining boundary-adjacent hard snippets and refining their representations with SniCo Loss. Experiments on THUMOS’14 and ActivityNet v1.2 validate its state-of-the-art performance.
- CoLA addresses the single snippet cheating problem in weakly-supervised action localization.
- Hard Snippet Mining locates hard snippets near action-instance boundaries.
- SniCo Loss refines mined hard-snippet representations using easy snippets from discriminative regions.
- Experiments on THUMOS’14 and ActivityNet v1.2 validate CoLA’s state-of-the-art performance.