A Hybrid Attention Mechanism for Weakly-Supervised Temporal Action Localization

Introduction

• WTAL localizes and classifies action instances using only video-level category labels during training, reducing the costly, time-consuming, and error-prone temporal annotation required by full supervision.Source: Introduction · p. 1
• MIL-based WTAL can capture only the most discriminative parts of an action, omitting less discriminative portions and the full temporal extent.Because training minimizes video-level classification loss, snippet-level class activation scores favor discriminative frames; Figure 1 illustrates this limitation for diving.Source: Introduction · p. 1, Figure 1 · p. 1
• Without effective background modeling, WTAL may include background frames in predictions, producing false positives; HAM-Net uses attention scores guided by an auxiliary background class.The introduction notes that background activity plays an important role in action localization (Lee, Uh, and Byun 2020).Source: Introduction · pp. 1–2
• HAM-Net combines soft, semi-soft, and hard attention to model actions in their entirety and background activity.The framework uses a classification branch for action and background class activation scores and an attention branch for snippet-level “action-ness” scores.Source: Introduction · p. 2
• Semi-soft and hard attention suppress more discriminative snippets to focus on less discriminative frames, encouraging the model to learn full action boundaries.Semi-soft attention zeroes snippets whose soft-attention score exceeds a threshold, while hard attention assigns attention scores of one to less discriminative portions.Source: Introduction · p. 2
• HAM-Net reports gains of 2.2% mAP at IoU threshold 0.5 on THUMOS14 and 1.3% mAP at IoU threshold 0.75 on ActivityNet1.2 over state-of-the-art methods.Source: Introduction · p. 2

Action Analysis with Full Supervision

• Fully supervised action analysis uses optical-flow two-stream networks and 3D convolutions for video representations, while temporal localization methods often use two-stage strategies.Video action recognition advanced with deep learning and large-scale datasets; the two-stream network (Simonyan and Zisserman 2014) incorporates optical flow, and 3D convolutional networks include Carreira and Zisserman (2017) and Tran et al. (2015, 2020). Two-stage localization methods include Tran et al. (2020), Zhao et al. (2017), Chao et al. (2018), and Lin et al. (2018).Source: Action Analysis with Full Supervision · p. 2

Weakly Supervised Temporal Action Localization

• Weakly supervised localization methods use varied selection, proposal, attention, and learning strategies, but prior approaches do not explicitly model an action instance in its entirety.UntrimmedNets (Wang et al. 2017), STPN (Nguyen et al. 2018), AutoLoc, W-TALC (Paul, Roy, and Roy-Chowdhury 2018), and Islam and Radke (Islam and Radke 2020) introduced selection, sparsity, boundary, and metric-learning strategies. Later work includes MAAN (Yuan et al. 2019), TSM, WSGN (Fernando, Tan, and Bilen 2020), DGAM (Shi et al. 2020a), CleanNet (Liu et al. 2019), and 3C-Net (Narayan et al. 2019); BaS-Net (Lee, Uh, and Byun 2020) and Nguyen et al. (Nguyen, Ramanan, and Fowlkes 2019) model background with an auxiliary class.Source: Weakly Supervised Temporal Action… · p. 2
• Action-completeness methods recover missing parts by hiding video segments or predicting distinctive parts, whereas HAM-Net hides the most discriminative segments.Hide-and-Seek (Singh and Lee 2017) hides part of the video to discover relevant parts; Liu et al. (Liu, Jiang, and Wang 2019) use a multi-branch network to predict distinctive action parts.Source: Weakly Supervised Temporal Action… · p. 2

Problem Formulation

• Training uses video-level action-presence labels without instance counts or order, while evaluation predicts each action’s start and end frames, label, and activity score.Source: Problem Formulation · pp. 2–3

Proposed Method

• HAM-Net combines classification and attention branches, using soft, semi-soft, and hard attention to guide class scores and four attention-guided losses alongside sparsity and guide losses.The attention-guided losses are BCL, SAL, SSAL, and HAL.Source: Figure 2 · p. 3

Snippet-Level Classification

• HAM-Net divides videos into non-overlapping snippets, extracts RGB and flow features, and concatenates them to represent each snippet’s appearance and motion cues.Snippet-level features enable use of 3D convolutional feature extractors that model temporal dependencies.Source: Snippet-Level Classification · p. 3
• The classification branch outputs snippet-level Class Activation Sequences for each action class plus a background class.Source: Snippet-Level Classification · p. 3
• Video-level class scores are formed by aggregating each class’s temporal top-k snippet scores and applying softmax across classes.Source: Snippet-Level Classification · p. 3
• The base classification loss is cross-entropy between ground-truth and predicted video-level class scores.Source: Snippet-Level Classification · p. 3
• Because background has no negative training samples, classification alone cannot learn it effectively; HAM-Net’s attention branch therefore explores each segment’s action-ness score.Source: Snippet-Level Classification · p. 3

A Hybrid Attention Mechanism for Weak Supervision

• HAM-Net uses a separate foreground–background attention module because motion features can reveal background regions and learning two classes is easier than many weakly supervised action classes.The module aims to assign lower attention to snippets without activity and higher attention to other regions.Source: A Hybrid Attention Mechanism for Weak… · p. 4

Soft Attention Score

• Soft attention assigns each snippet a foreground score and uses it to weight class logits, creating foreground-focused scores that suppress the background class.The attention score is produced from snippet features by two temporal convolution layers followed by a sigmoid.Source: Soft Attention Score · p. 4

Semi-Soft Attention Score

• Semi-soft attention thresholds soft attention to drop the most discriminative regions while retaining foreground snippets, so its video-level scores contain foreground activities only.Its loss treats the background label as absent because semi-soft attention suppresses background snippets as well as the most discriminative regions.Source: Semi-Soft Attention Score · p. 4

Hard Attention Score

• Hard attention weights the original snippet logits to form video-level scores, removing the more discriminative regions while retaining background snippets.Accordingly, its loss uses the ground-truth label with background activity.Source: Hard Attention Score · p. 4

Loss Functions

• HAM-Net’s joint loss includes sparsity and guide losses: sparsity penalizes soft-attention scores, while guide loss aligns background-class probability with background attention.The guide loss minimizes the absolute difference between the background probability from class activation logits and 1 − ai.Source: Loss Functions · pp. 4–5

Temporal Action Localization

• Classes with video-level scores below 0.1 are discarded, then soft-attention thresholding removes background snippets and connected components form class-agnostic proposals.The class-specific scores use attention-guided class logits.Source: Temporal Action Localization · p. 5
• Candidate action locations are scored using the outer-inner score of AutoLoc.The score uses video-level class scores and snippet-level class logits.Source: Temporal Action Localization · p. 5
• The method applies different thresholds to generate action proposals and removes overlapping segments with non-maximum suppression.Source: Temporal Action Localization · p. 5

Datasets

• THUMOS14 has 20 action categories and about 15.5 activity segments per video, while ActivityNet1.2 has 200 classes and about 1.5 instances per video.THUMOS14 has 200 validation and 213 test videos; ActivityNet1.2 has 4,819 training and 2,382 test videos.Source: Datasets · p. 5

Evaluation Metrics

• Evaluation uses the standard protocol and reports mean Average Precision (mAP) at multiple intersection over union (IoU) thresholds.The ActivityNet evaluation code is used to calculate the metrics.Source: Evaluation Metrics · p. 5

Implementation Details

• The method uses concatenated 2048-dimensional RGB and flow I3D features, with dataset-specific training snippet counts, optimizer schedules, and localization thresholds.Features come from non-overlapping 16-frame chunks; Adam is used at learning rate 0.00001 for 100 THUMOS14 epochs and 20 ActivityNet epochs. Localization thresholds range from 0.1 to 0.9 in 0.05 steps, followed by non-maximum suppression.Source: Implementation Details · p. 5

Ablation Studies

• The ablations indicate that all loss components are required for maximum performance, and the final approach captures full action boundaries where earlier variants do not.In the Long Jump visualization, MIL-only predicts false positives; sparsity and guide losses remove them, but the full boundaries appear with the proposed approach.Source: Ablation Studies · p. 5, Figure 3 · p. 6

Importance of sparsity and guide loss

• Adding sparsity and guide losses yields 4% mAP gain in MIL-only mode and 9% in MIL-and-Drop mode.Without sparsity, soft-attention scores mostly remain near 1, making snippet dropping ineffective; guide loss works best in combination with sparsity.Source: Importance of sparsity and guide loss · pp. 5–6
• Adding LAL raises THUMOS14 AVG mAP from 34.8 to 39.8.AVG mAP averages mAP values across IoU thresholds 0.1 to 0.7.Source: Table 1 · p. 6

Importance of attention losses

• 6.2% average mAP gain over BCL-only came from LSAL; combining SSAL and HAL improved performance by 5% over the best MIL-only score and helped capture full action instances.The MIL-only model missed parts of Long Jump instances, while adding attention losses helped capture the action in its entirety.Source: Importance of attention losses · p. 6

Importance of dropping snippets by selective thresholding

• Selective dropping of the most discriminative snippets outperforms random dropping and dropping less discriminative snippets, while inverse dropping decreases localization performance.The comparison supports selectively dropping discriminative foreground snippets to train on less discriminative video parts.Source: Importance of dropping snippets by… · p. 6, Figure 4 · p. 6

Ablation on λ2 and λ3

• When λ2 = λ3 = λ, average mAP is 34.8% at λ = 0, improves until λ = 0.2, then degrades; 0.2 balances insufficient attention to LSSAL/LHAL against overemphasis on less discriminative parts.The passage attributes degradation at higher weights to potentially ignoring more discriminative regions in each iteration, harming localization.Source: Ablation on λ2 and λ3 · p. 6

Performance Comparison to State-of-the-Art

• 2.2% more mAP than the current best scores at IoU 0.5; HAM-Net also achieves state-of-the-art scores on all THUMOS14 IoU thresholds and outperforms WTAL approaches across ActivityNet1.2 metrics.On THUMOS14, HAM-Net also outperforms some fully supervised methods and is comparable to some recent fully supervised methods.Source: Table 2 · p. 7, Performance Comparison to… · p. 7, Table 3 · p. 7

Qualitative Performance

• HAM-Net captures the full temporal extent of activities, whereas the model trained without LHAL and LSSAL focuses only on more discriminative snippets.Figure 5 shows representative THUMOS14 examples with ground-truth localization, HAM-Net predictions, and predictions without HAL and SSAL.Source: Qualitative Performance · p. 7, Figure 5 · p. 7

• HAM-Net achieves state-of-the-art performance on both the THUMOS14 and ActivityNet1.2 datasets.Source: Conclusion · p. 7
• HAM-Net learns temporal action localization using only video-level supervision during training.Source: Conclusion · p. 7
• Its soft, semi-soft, and hard attentions distinguish background from foreground frames and capture full action boundaries.Source: Conclusion · p. 7

Action Classification

• 7.2% classification mAP improvement over 3C-Net on THUMOS14 makes HAM-Net competitive with state-of-the-art methods.The paper also describes its ActivityNet1.2 classification performance as very competitive, although HAM-Net was not designed for action recognition.Source: Action Classification · p. 10

Detailed Performance on ActivityNet1.2

• ActivityNet1.2 localization performance is reported as mAP at different IoU thresholds.The supplied passage gives no threshold-specific values.Source: Detailed Performance on ActivityNet1.2 · p. 10

More Ablation

• Without sparsity loss the model hardly learns localization, and performance rises with α, reaching its best score at α = 0.8.AVG mAP is the mean mAP from IoU thresholds 0.1 to 0.7 in increments of 0.1.Source: More Ablation · p. 10
• The guide-loss weight β = 0.8 gives the best performance, while γ is evaluated across different dropping-threshold values.Source: More Ablation · p. 10
• Larger training video lengths improve performance on THUMOS14, whereas ActivityNet performs best with 80 segments.The paper notes that THUMOS14 videos contain denser activity than ActivityNet videos.Source: More Ablation · p. 10
• Segments are chosen randomly only during training, while all segments are used during evaluation.Source: More Ablation · p. 10

More Qualitative Examples

• HAM-Net captures most of the several Pole Vault occurrences shown in the qualitative examples.The examples are from THUMOS14.Source: More Qualitative Examples · p. 10, Figure 7 · p. 12
• The model sometimes predicts high jump for attempts that resemble a jump but stop before the full action is completed, a failure also reported in prior WTAL methods (Islam and Radke 2020; Paul, Roy, and Roy-Chowdhury 2018).The paper shows these failure examples in Fig. 7b and Fig. 7c.Source: More Qualitative Examples · p. 10, Figure 7 · p. 12