Source-linked AI summary
W-TALC: Weakly-supervised Temporal Activity Localization and Classification
Sujoy Paul, Sourya Roy, Amit K Roy-Chowdhury
TL;DR
Frame-wise activity annotations are costly, motivating temporal activity localization and classification from video-level weak labels. W-TALC combines a Two-Stream feature extractor with a weakly supervised module using jointly optimized Multiple Instance Learning and Co-Activity Similarity losses. Experiments on Thumos14 and ActivityNet1.2 report state-of-the-art results while processing long videos at high temporal granularity.
Problem
Frame-wise annotations for temporal activity localization are labor-intensive, motivating models that learn from easier-to-obtain video-level labels.
Method
W-TALC uses off-the-shelf Two-Stream features and jointly optimizes Multiple Instance Learning and Co-Activity Similarity losses using video-level labels.
Results
Experiments on Thumos14 and ActivityNet1.2 report state-of-the-art results for weak temporal activity localization and classification.
Takeaways & Limitations
W-TALC detects activities at high temporal granularity in long videos without fine-tuning the feature extractor.
Takeaways & Limitations
CASL cannot be computed when a video contains an activity but has only one feature vector, because no low-attention feature is defined.
Abstract
from arXiv · showhide
Most activity localization methods in the literature suffer from the burden of frame-wise annotation requirement. Learning from weak labels may be a potential solution towards reducing such manual labeling effort. Recent years have witnessed a substantial influx of tagged videos on the Internet, which can serve as a rich source of weakly-supervised training data. Specifically, the correlations between videos with similar tags can be utilized to temporally localize the activities. Towards this goal, we present W-TALC, a Weakly-supervised Temporal Activity Localization and Classification framework using only video-level labels. The proposed network can be divided into two sub-networks, namely the Two-Stream based feature extractor network and a weakly-supervised module, which we learn by optimizing two complimentary loss functions. Qualitative and quantitative results on two challenging datasets - Thumos14 and ActivityNet1.2, demonstrate that the proposed method is able to detect activities at a fine granularity and achieve better performance than current state-of-the-art methods.
1 Introduction
W-TALC addresses temporal activity localization and classification using video-level weak labels instead of frame-wise annotations. It combines a Two-Stream feature extractor with a weakly supervised module that jointly optimizes Multiple Instance Learning and Co-Activity Similarity losses.
- Frame-wise activity annotation is labor-intensive and difficult to scale, whereas video-level tags provide a weaker but easier supervision source.
- At test time, W-TALC estimates video activity labels and localizes each detected activity by start time, end time, category, and recognition confidence.
- W-TALC uses off-the-shelf Two-Stream networks as feature extractors and learns task-specific fully connected and label projection layers.The feature extractor is not fine-tuned for the weakly supervised task.
- Multiple Instance Learning Loss pools class-wise temporal activations, while Co-Activity Similarity Loss uses attention-derived weak temporal labels and pair-wise video constraints.The two losses are jointly minimized to learn the network parameters.
- The framework does not require activity ordering during training and can detect multiple activities occurring within the same temporal duration.
2 Related Works.
Prior work uses weak supervision for spatial localization, scripts, activity ordering, and untrimmed video classification, but these settings often require additional information. W-TALC targets weak temporal localization without ordering labels while addressing long-video granularity through pair-wise constraints.
- Weakly-supervised Spatial Action Localization: Weak supervision has been applied to spatial actor localization, often using fully supervised person detectors or generated person tubes.
- Scripts as Weak Supervision: Scripts and subtitles provide weak temporal or spatial action supervision by supplying action words or actor-action pairs.
- Temporal Localization with Ordering: Several temporal-localization methods use activity ordering during training, but such ordering may be unavailable for many web videos.
- Related approaches use fixed video segments, sparsity, or temporal supervision, whereas W-TALC uses pair-wise constraints and a mechanism for long videos at high temporal granularity.The paper states that CASL is complementary to MILL and that W-TALC achieves better performance than state-of-the-art weak TALC methods without fine-tuning the feature extractor.
3 Methodology
W-TALC learns weakly supervised temporal activity localization and classification from video-level labels. It combines two-stream features with label-space activations, MIL-based classification, and co-activity similarity constraints while managing long-video computation.
- The framework extracts temporal features with two-stream UntrimmedNet or I3D architectures before applying a learned weakly supervised module.The framework is designed to remain agnostic to the selected feature representation.
- Long videos create a performance-computation trade-off: fine temporal processing improves granularity but increases memory and computation requirements.The method processes short videos fully and samples contiguous clips of length T from longer videos.
- W-TALC uses only video-level labels to learn simultaneous activity localization and classification, without assuming temporal label ordering.
- A label-space projection produces class-wise activations at temporal instants, which represent the possible occurrence of activities over time.The projection uses a fully connected layer with weights shared along the temporal axis.
- Multiple Instance Learning Loss pools class-wise activations with a k-max-mean strategy, forms a category probability distribution, and compares it with normalized video labels using cross-entropy.Videos may contain multiple activities, so their label vectors are normalized before computing the loss.
- Co-Activity Similarity Loss encourages videos sharing an activity category to learn similar representations, addressing relationships not directly enforced by MILL.It uses class-wise activations to identify activity portions and jointly learns feature representations and label-space projection.
4 Experiments
Experiments evaluate W-TALC for weakly supervised activity localization and classification on ActivityNet1.2 and Thumos14, including dataset setup, comparative performance, loss weighting, sequence-length sensitivity, and qualitative detections.
- Datasets: Experiments use ActivityNet1.2 and Thumos14, retaining only video-level activity tags despite the datasets providing frame-wise annotations.ActivityNet1.2 has 100 classes, while Thumos14 includes 20 temporally annotated categories with multiple activities and widely varying video lengths.
- Implementation Details: W-TALC uses fixed feature extractors and trains weakly supervised layers on a single Tesla K80 GPU with TensorFlow.Features come from UntrimmedNets and I3D, and the feature extractors are not fine-tuned.
- Activity Localization: The framework performs better than other weakly supervised methods with similar feature usage for temporal activity localization on Thumos14 and ActivityNet1.2.The ActivityNet1.2 evaluation is presented as the first quantitative result on weakly supervised temporal activity localization there, according to the authors.
- Activity Classification: The proposed method performs significantly better than other state-of-the-art approaches for activity classification on Thumos14 and ActivityNet1.2.Some comparison methods use larger training sets, as indicated in the classification tables.
- Relative Weights on Loss Functions: Detection is best at λ = 0.5, where MILL and CASL receive equal weights; using only MILL decreases mAP by 7−8%.The experiment uses I3D features on Thumos14 and varies λ, with higher values assigning more weight to MILL.
- Sensitivity to Maximum Length of Sequence: Increasing the maximum training sequence length initially improves detection while increasing computation, with performance eventually plateauing at T = 320s.The sampling strategy enables learning from long videos and reduces computation at lower T, but introduces training-label errors that can increase with shorter sampling limits.
5 Conclusions and Future Work
W-TALC learns temporal activity localization and video classification from video-level labels, introducing Co-Activity Similarity loss alongside Multiple Instance Learning. A mechanism supports high-granularity processing of long videos, and experiments on two challenging datasets achieve state-of-the-art weak TALC results.
- W-TALC learns temporal activity localization and video classification using only weak supervision with video-level labels.
- Co-Activity Similarity loss is designed to complement the Multiple Instance Learning loss.The paper reports this complementarity empirically.
- A simple mechanism enables processing long videos at high temporal granularity.
- Experiments on two challenging datasets achieve state-of-the-art results for weak temporal activity localization and classification.