Source-linked AI summary
UntrimmedNets for Weakly Supervised Action Recognition and Detection
Limin Wang, Yuanjun Xiong, Dahua Lin, Luc Van Gool
TL;DR
The paper addresses the cost and ambiguity of obtaining temporally trimmed training videos by learning from untrimmed videos with only video-level labels. UntrimmedNet jointly performs clip-level classification and proposal selection end to end, achieving better recognition and comparable detection performance to strongly supervised methods on THUMOS14 and ActivityNet.
Problem
Temporal action annotations are expensive, time-consuming, and sometimes subjective, motivating learning from untrimmed videos using only video-level labels.
Method
UntrimmedNet combines a classification module with a selection module that classifies clip proposals and selects or ranks important proposals without temporal annotations.
Results
UntrimmedNet achieves better action-recognition and comparable action-detection performance than strongly supervised methods on THUMOS14 and ActivityNet.
Takeaways & Limitations
Jointly learning classification and selection end to end supports weakly supervised action recognition and detection from untrimmed videos.
Takeaways & Limitations
Uniform sampling can generate imprecise proposals because it ignores the continuous and consistent properties of action instances.
Abstract
from arXiv · showhide
Current action recognition methods heavily rely on trimmed videos for model training. However, it is expensive and time-consuming to acquire a large-scale trimmed video dataset. This paper presents a new weakly supervised architecture, called UntrimmedNet, which is able to directly learn action recognition models from untrimmed videos without the requirement of temporal annotations of action instances. Our UntrimmedNet couples two important components, the classification module and the selection module, to learn the action models and reason about the temporal duration of action instances, respectively. These two components are implemented with feed-forward networks, and UntrimmedNet is therefore an end-to-end trainable architecture. We exploit the learned models for action recognition (WSR) and detection (WSD) on the untrimmed video datasets of THUMOS14 and ActivityNet. Although our UntrimmedNet only employs weak supervision, our method achieves performance superior or comparable to that of those strongly supervised approaches on these two datasets.
1. Introduction
The paper introduces weakly supervised learning of action recognition and detection models directly from untrimmed videos using only video-level labels. UntrimmedNet jointly learns action classification and temporal proposal selection end to end, achieving strong results on two untrimmed-video benchmarks.
- Temporal annotation is costly, time-consuming, potentially subjective, and difficult to define consistently for action instances.
- Weakly supervised action recognition and detection learn from untrimmed videos using only video-level action labels, without temporal instance annotations.
- The learning problem requires both visual action modeling and automatic reasoning about possible action-instance locations.
- UntrimmedNet is an end-to-end architecture combining a classification module for action models with a selection module for detecting action instances.
- Clip proposals are generated by uniform or shot-based sampling, then classified individually while the selection module selects or ranks important proposals.
- Experiments on THUMOS14 and ActivityNet show superior recognition and comparable detection performance against strongly supervised state-of-the-art methods.
2. Related Work
The related work covers deep architectures for video action recognition and several forms of weak supervision. UntrimmedNet differs by using no temporal information and by applying its approach beyond movie videos, while its pipeline combines classification with proposal selection.
- Deep action-recognition research has explored convolutional and other deep network architectures for video understanding.
- Prior weakly supervised video methods use movie scripts, ordered action-class lists, or other supervision to learn or localize action instances.
- Unlike movie-script supervision, the paper's weak supervision provides no temporal information about action instances and applies to all kinds of videos.
- The UntrimmedNet pipeline samples short clips, extracts features, classifies each proposal, selects or ranks important proposals, and combines outputs for video-level prediction.
3. Learning from Untrimmed Videos
UntrimmedNet learns from untrimmed videos by generating clip proposals and jointly modeling clip classification with temporal proposal selection in an end-to-end architecture. Its differentiable modules support weakly supervised learning using only video-level labels.
- Clip proposal generation: UntrimmedNet generates short clip proposals from untrimmed videos before feature extraction and model training.The proposal strategies are uniform sampling and shot-based sampling.
- Clip proposal generation: Uniform sampling divides a long video into N equal-duration clips but can produce temporally imprecise proposals.This method assumes action instances may have relatively short durations and ignores their continuous, consistent properties.
- Clip proposal generation: Shot-based sampling detects shot changes using adjacent-frame HOG differences, samples fixed-duration clips within each shot, and merges the proposals.The method is designed around the expectation that an action instance focuses on consistent motion within a single shot.
- UntrimmedNet architecture: The architecture combines feature extraction, classification, and selection modules that are differentiable and trainable end to end.Feature representations φ(V; c) are extracted for each clip proposal and passed to later action-recognition layers.
- UntrimmedNet architecture: The classification module maps each clip representation to action-class scores, while the selection module identifies proposals likely to contain action instances.Selection uses either hard top-k multiple-instance learning or soft attention-based ranking.
- UntrimmedNet architecture: Classification softmax operates across action classes for each proposal, whereas selection softmax compares proposals for the same action class.The distinction separates class prediction from temporal proposal weighting.
- Video-level prediction: Video-level prediction combines classification and selection scores by averaging selected top-k proposals for hard selection or weighted summation for soft selection.Soft selection uses normalized classification scores and attention weights, producing a convex combination without further normalization.
4. Action Recognition and Detection
The learned UntrimmedNet models are used for recognition and temporal detection in untrimmed videos. Recognition aggregates snippet predictions, while detection uses soft-selection attention weights to suppress background and localize actions.
- Action recognition: UntrimmedNet performs action recognition as snippet-wise evaluation on untrimmed videos.It samples a frame or stacked optical-flow frames every 30 frames and aggregates scores with top-k pooling.
- Action detection: The soft-selection model supports temporal action detection because it outputs an attention weight for each snippet.For more precise localization, predictions and attention weights are evaluated every 15 frames, and background is removed by thresholding attention.
5. Experiments
Experiments on THUMOS14 and ActivityNet evaluate UntrimmedNet’s configurations and compare its weakly supervised recognition and detection performance with existing methods. The studies examine sampling, feature extraction, proposal counts, selection modules, and qualitative attention behavior.
- Datasets and setup: Experiments use THUMOS14 and ActivityNet, which provide untrimmed videos; temporal annotations are not used to train UntrimmedNets.THUMOS14 training uses 1,010 validation videos and evaluation uses 1,574 test videos.
- Exploration studies: 71.6% versus 70.2% shows shot-based sampling outperforming uniform sampling for soft selection on THUMOS14.Both sampling methods provide good performance, and shot-based sampling is selected by default afterward.
- Exploration studies: Temporal Segment Networks consistently outperform two-stream CNNs for both hard and soft selection, attributed to long-term modeling over the entire clip.The comparison uses seven clip proposals per video during training.
- Exploration studies: For soft selection, seven and nine proposals show a small advantage over five, while two-stream networks remain stable across proposal counts with hard selection.The proposal-count study balances sampled clips per video against the number of videos per batch under limited GPU memory.
- Weakly supervised action recognition: UntrimmedNet with hard or soft selection outperforms original TSN frameworks on both datasets, and remains above TSN with 21 segments.The 21-segment TSN matches UntrimmedNet’s 3×7 segment count, isolating the contribution of explicit selection modules.
6. Conclusions
UntrimmedNet directly learns action models from untrimmed videos for weakly supervised recognition and detection. Across two challenging datasets, it achieves better or comparable performance to strongly supervised methods, which the authors associate with jointly designed, end-to-end optimized classification and selection modules.
- UntrimmedNet directly learns action models from untrimmed videos for weakly supervised action recognition and detection.The architecture is designed for both tasks without temporal annotations of action instances.
- On two challenging untrimmed-video datasets, UntrimmedNet achieves better or comparable performance than strongly supervised methods.The conclusion reports this pattern for both action recognition and detection.
- The authors ascribe UntrimmedNet’s superior performance to jointly designing classification and selection modules and optimizing them end-to-end.