Source-linked AI summary
Learning to track for spatio-temporal action localization
Philippe Weinzaepfel, Zaid Harchaoui, Cordelia Schmid
TL;DR
The paper tackles spatio-temporal action localization, where actions must be localized spatially and temporally in realistic videos. It detects and scores frame-level proposals, tracks high-scoring candidates with instance- and class-level detectors, scores tracks with CNN and STMH features, and applies track-level temporal windows. It outperforms the state of the art by 15% in mAP on UCF-Sports, 7% on J-HMDB, and 12% on UCF-101.
Problem
Spatio-temporal action localization must determine both the spatial and temporal extent of actions, which is challenging when actions vary in duration and per-frame localization is uncertain.
Method
The method scores frame-level proposals with static and motion CNN features, tracks selected proposals using instance- and class-level detectors, scores tracks with CNN and STMH features, and localizes duration with sliding windows.
Results
15% in mAP on UCF-Sports, 7% on J-HMDB, and 12% on UCF-101 over the state of the art.
Takeaways & Limitations
Tracking and spatio-temporal track descriptors support precise spatial localization and temporal action localization at the track level.
Abstract
from arXiv · showhide
We propose an effective approach for spatio-temporal action localization in realistic videos. The approach first detects proposals at the frame-level and scores them with a combination of static and motion CNN features. It then tracks high-scoring proposals throughout the video using a tracking-by-detection approach. Our tracker relies simultaneously on instance-level and class-level detectors. The tracks are scored using a spatio-temporal motion histogram, a descriptor at the track level, in combination with the CNN features. Finally, we perform temporal localization of the action using a sliding-window approach at the track level. We present experimental results for spatio-temporal localization on the UCF-Sports, J-HMDB and UCF-101 action localization datasets, where our approach outperforms the state of the art with a margin of 15%, 7% and 12% respectively in mAP.
1. Introduction
Spatio-temporal action localization remains challenging because actions vary in temporal extent and must be localized spatially as well as temporally. The paper addresses this by tracking frame-level proposals and combining CNN and spatio-temporal descriptors for track-level localization.
- Motivation: Action localization must identify both where and when an action occurs, unlike recognition methods that typically classify cropped videos.Actions can have varying temporal extents, making localization in long video streams necessary.
- Related challenges: Prior work often emphasizes either spatial localization in images or temporal localization in videos rather than solving both jointly.Related approaches use local detectors for spatial localization or dense motion features for temporal localization.
- Motivation: Uncertain per-frame localization makes temporal consistency difficult to enforce directly.The paper uses proposals for per-frame spatial candidates and a tracker combining instance-level and class-level detectors.
- Approach: The approach scores frame-level proposals with appearance and motion CNN descriptors, tracks them, and then scores tracks using CNN features and a spatio-temporal motion histogram.Temporal localization is subsequently performed at the track level.
- Contribution: The paper reports state-of-the-art spatio-temporal localization results on UCF-Sports, J-HMDB, and UCF-101.A spatio-temporal local descriptor helps select relevant tracks and temporally localize actions.
2. Related work
Related work primarily addresses action classification, spatial or temporal localization, and joint spatio-temporal localization through models, parts, or proposals. The paper distinguishes its approach by tracking proposals without forcing detections through proposals at every frame and by adding track-level spatiotemporal descriptors.
- Action recognition: Most action-recognition work focuses on assigning a category label to a video using space-time descriptors or CNN-based representations.Approaches include 3D convolutions, recurrent networks, and separate processing of images and optical flow.
- Temporal localization: Temporal action localization commonly uses sliding windows, with some work replacing them through more efficient search strategies.Sliding windows are also used for spatio-temporal localization in approaches extending deformable part models.
- Figure-centric models: Figure-centric models localize actions by modeling visual words, human-object interactions, or detected humans.These methods treat subject location as latent or explicitly detect people and objects.
- Action proposals: Action-proposal methods generate candidate regions using actionness scores, supervoxel hierarchies, or discriminative parts.These proposals support spatio-temporal action localization.
- Positioning: The paper links object proposals across video frames but does not force detections to pass through proposals at every frame.It also combines frame-level CNN features with spatiotemporal descriptors to capture action dynamics.
3. Overview of the approach
The approach proceeds from frame-level proposal detection and scoring to proposal tracking, track scoring, and temporal localization. It combines CNN appearance and motion information with instance- and class-level tracking and a spatio-temporal motion histogram.
- Overview: The pipeline has four stages: frame-level proposal extraction and scoring, tracking, track scoring, and temporal localization.The stages are presented as the overview of the spatio-temporal action-localization approach.
- Extracting and scoring frame-level proposals: EdgeBoxes produces high-recall frame-level candidates, which are scored using CNN features encoding static appearance and motion information.The method keeps the best 256 EdgeBoxes proposals per frame and trains classifiers to distinguish actions from background.
- Tracking best candidates: Highest-scoring proposals are tracked throughout the video with tracking-by-detection using both instance-level and class-level information.The tracker is applied repeatedly for each action starting from high-scoring proposals.
- Scoring tracks: Tracks are scored with CNN features and a spatio-temporal motion histogram that captures action dynamics across multiple frames.The STMH computes histograms of gradient, optical flow, and motion boundaries over spatio-temporal cells in fixed-length track chunks.
- Temporal localization: A multi-scale sliding window over each track selects the temporal window with the highest score using CNN features, STMH, and a learned duration prior.At test time, windows of different lengths are shifted by 10 frames.
4. Detailed description of the approach
The approach detects and scores frame-level proposals, tracks selected regions using instance- and class-level detectors, scores tracks with spatio-temporal descriptors, and temporally localizes actions with sliding windows.
- Overview: The task is to output action regions across a predicted temporal extent, identifying when and where class c appears in a video.
- Frame-level proposals: Each frame supplies up to 256 EdgeBoxes proposals, represented with CNN features combining static RGB appearance and optical-flow motion information.The spatial-CNN captures appearance, while the motion-CNN captures motion patterns; their fc7 outputs are concatenated.
- Tracking: The tracker selects top-scoring action classes and links proposals across frames using both class-level and instance-level detectors.It generates tracks from high-scoring proposals and can remain robust to missing proposals during tracking.
- Track descriptor: STMH divides 15-frame track chunks into spatiotemporal cells and computes histograms of gradient, optical flow, and motion boundaries.Using 3 temporal cells and 8 × 8 spatial cells produces a 6336-dimensional descriptor.
- Scoring and temporal localization: Track scores combine frame-level CNN scores with chunk-level descriptors, while temporal localization selects the highest-scoring window from multiple track-level window lengths.Windows are slid with a stride of 10 frames and scored using CNN features, STMH, and a learned duration prior.
5. Experimental results
The experiments evaluate the approach on three datasets, analyze tracker, class-selection, and STMH design choices, and compare performance with prior work. The method outperforms the state of the art across UCF-Sports, J-HMDB, and UCF-101.
- Experimental setup: Experiments use UCF-Sports, J-HMDB, and UCF-101 with the paper’s stated evaluation protocol.The section studies datasets, tracking and class selection, STMH parameters, and state-of-the-art comparisons.
- Impact of the tracker: Combining instance-level and class-specific detectors gives the best final detection performance on UCF-Sports and J-HMDB.On UCF-Sports, the combination mainly improves recall; on J-HMDB, instance-only tracking raises recall but lowers precision through high-scoring incorrect-label tracks.
- Impact of the tracker: 6% gain comes from EdgeBoxes over SelectiveSearch with STMH, followed by a further 7% gain from tracking on UCF-Sports localization mAP.The comparison also reports more refined localization with tracking, identifying the tracker as a key component.
- STMH analysis: Ns = 8 spatial cells gives the best STMH classification performance on both datasets, independently of Nt.The study evaluates mean Accuracy on ground-truth tracks and compares STMH with improved dense trajectories, whose reported performance is 91.9% on UCF-Sports and 57.99% on J-HMDB.
- Comparison to the state of the art: The approach outperforms the state of the art by 15% on UCF-Sports with 90.5% mAP, by more than 7% on J-HMDB, and achieves over 47% mAP on UCF-101 at δ = 20%.On UCF-101 at δ = 5%, it obtains 54% mAP versus 42% reported by.
6. Conclusion
The approach detects actions in space and time by tracking frame-level proposals and combining CNN and spatio-temporal descriptors before temporal localization. It improves state-of-the-art mAP by 15% on UCF-Sports, 7% on J-HMDB, and 12% on UCF-101.
- The pipeline tracks frame-level object proposals throughout each video using instance-level and class-level detectors.Tracks are scored with CNN features and spatio-temporal descriptors, then temporally localized with a sliding window.
- 15% mAP improvement over the state of the art is reported on UCF-Sports.
- 7% mAP improvement over the state of the art is reported on J-HMDB.
- 12% mAP improvement over the state of the art is reported on UCF-101.