Adversarial Background-Aware Loss for Weakly-supervised Temporal Activity Localization

1 Introduction

• Weakly supervised localization assumes only video-level activity labels are available, which are easier to collect than frame-level boundary annotations.The paper describes these labels as available across many datasets, allowing application in a broader range of situations.Source: §1 · p. 1
• Existing weakly supervised methods struggle to distinguish background information from activities of interest, although this ability is critical to strong temporal localization.Prior approaches include background suppression and background-content modeling.Source: §1 · p. 2
• The first triplet aggregates embedded features using top-down attention into a video-level feature and groups features of the same activity while encouraging high attention where the activity occurs.It adapts angular triplet-center loss to videos with variable numbers of frames.Source: §1 · p. 3
• The second triplet uses a background-attended video-level feature and the activity-attended feature with their center, making the original feature closer to its center than the background-focused one.The paper calls the paired-triplet approach ACL-PT and the adversarial two-branch final method A2CL-PT.Source: Figure 1 · p. 2, §1 · p. 3
• A2CL-PT combines two triplets with two adversarial branches to distinguish background from activity features and localize activities more thoroughly.One triplet learns discriminative features; the other distinguishes background features from activity-related features, while the branches seek supplementary activities from video regions left by the first branch.Source: §1 · pp. 3–4
• The proposed method uses paired triplets and an adversarial two-branch network to localize more complete activities, and experiments on THUMOS14 and ActivityNet show it outperforms previous state-of-the-art approaches.Source: §1 · p. 4

2 Related Work

• Angular triplet-center loss improves triplet-center loss by using angular distance, whose geometric interpretation makes its margin easier to design and bounds it from 0 to π.Center loss reduces intra-class variation, while triplet-center loss increases inter-class separability using a nearest negative center.Source: §2 · p. 4
• BaS-Net and Nguyen et al. use background portions, while Liu et al. uses a diversity loss to differentiate multi-branch T-CAMs.The paper also notes that 3C-Net applies center loss but is limited because it does not consider inter-class separability.Source: §2 · p. 4
• The framework uses RGB and optical-flow streams with first and adversarial branches; pre-trained I3D features are embedded through two ReLU fully connected layers, then T-CAMs are computed using 1-D convolutions.The first branch’s most salient activity features are zeroed out for each activity class, and the resulting features are applied with different 1-D convolutional layers. The input streams use pre-trained I3D networks [1].Source: Figure 2 · p. 5
• A2CL-PT adapts adversarial complementary learning, which erases salient regions for a second branch, by merging that idea with center loss for temporal activity localization.The related-work passage identifies ACoL as a weakly supervised object-localization method.Source: §2 · p. 5

3.1 Feature Embedding

• Video segments are embedded from RGB and optical-flow streams using two fully connected layers with ReLU after I3D features are extracted from non-overlapping 16-frame segments.The pre-extracted features have dimension D = 1024.Source: §3.1 · p. 6

3.2 Angular Center Loss with a Pair of Triplets (ACL-PT)

• A 1-D convolution maps embedded features to T-CAM class scores over time, and class-wise attention aggregates features into video-level representations.The original attention is treated as a probability mass function over time for each activity class.Source: §3.2 · p. 6
• The paired triplets group video-level features by activity class and use a second attention to emphasize background features over activity features.The new attention is derived from T-CAM and is intended to have lower activity-feature values and higher background-feature values than the original attention.Source: §3.2 · p. 7

3.3 Adopting an adversarial approach (A2CL-PT)

• Two adversarial branches sum their ACL-PT losses into A2CL-PT, with the second branch using features from salient regions zeroed out to find supplementary activities.The zeroed-out feature ratio is controlled by hyperparameter s_a.Source: §3.3 · p. 8

3.4 Classification Loss

• The classification objective uses cross-entropy between predicted and ground-truth activity probability distributions, with scores formed by averaging top-k T-CAM values.The final T-CAM combines the RGB and optical-flow streams and their adversarial branches using learned class-specific weights.Source: §3.4 · p. 9

3.5 Classification and Localization

• At test time, positive classification scores select candidate classes, and successive positive T-CAM values define non-overlapping temporal segments scored by peak T-CAM plus class score.Only runs of at least two successive time steps are localized.Source: §3.5 · p. 9

4.1 Datasets and Evaluation

• THUMOS14 validation videos train the model without temporal boundaries, while ActivityNet1.3 uses its training set for training and validation set for evaluation.The evaluation reports mAP at different intersection-over-union thresholds.Source: §4.1 · p. 10

4.2 Implementation Details

• The experiments use I3D features from non-overlapping 16-frame RGB and optical-flow segments, with dataset-specific training settings.For ActivityNet1.3, an additional 1-D convolutional layer post-processes the final T-CAM; the reported training runs use 40.5k and 175k iterations on THUMOS14 and ActivityNet1.3, respectively.Source: §4.2 · p. 10

4.3 Comparisons with the State-of-the-art

• A2CL-PT gains 2.1% in average mAP over BaS-Net on THUMOS14 and achieves the best performance on ActivityNet1.3.On THUMOS14, it also outperforms weakly supervised methods across all reported metrics and weakly supervised methods using additional data at higher IoUs.Source: Table 1 · p. 11, §4.3 · p. 11, Table 2 · p. 12

4.4 Ablation Study and Analysis

• A2CL-PT performs best in the ablation study, with both the new triplet and adversarial approach contributing to the best performance.Adding an adversarial branch alone brings no gain without the new triplet; ACL-PT increases localization performance but also makes the network sensitive to salient activity features.Source: §4.4 · pp. 11–12
• Positive α improves performance, while ω greater than 1 makes adversarial T-CAMs dominant rather than supplementary.The reported performance increases with the adversarial approach when ω is less than or equal to 1.Source: §4.4 · p. 12, Figure 3 · p. 13

4.5 Qualitative Analysis

• Qualitative examples show lookalike-driven false positives, annotation-related false negatives, and successful localization of all HighJump instances.The reported examples include javelin swings mistaken for throws, a person drawing a field line mistaken for HammerThrow, and ThrowDiscus segments annotated despite the activity not occurring.Source: §4.5 · pp. 12–13

• A2CL-PT uses two feature-space triplets to learn discriminative features and distinguish background from activity-related video portions, alongside adversarial branches for more thorough localization.The conclusion reports that experiments show effectiveness, that the method outperforms existing methods on major datasets, and that both contributions are significant in ablation studies.Source: §5 · p. 13