Source-linked AI summary
Action Tubelet Detector for Spatio-Temporal Action Localization
Vicky Kalogeiton, Philippe Weinzaepfel, Vittorio Ferrari, Cordelia Schmid
TL;DR
Spatio-temporal action localization commonly detects actions frame by frame and links them, although single frames can be ambiguous. ACT-detector processes short frame sequences with SSD-style anchor cuboids to output tubelets. It improves scoring and localization and achieves state-of-the-art frame-mAP and video-mAP on J-HMDB and UCF-101, especially at high overlap thresholds.
Problem
Frame-level action localization does not exploit temporal continuity, although single-frame appearance and motion can be ambiguous.
Method
ACT-detector extends SSD with anchor cuboids, stacking shared convolutional features from a fixed-length frame sequence to score and regress tubelets.
Results
ACT-detector achieves state-of-the-art frame-mAP and video-mAP on J-HMDB and UCF-101, with particularly strong performance at high overlap thresholds.
Takeaways & Limitations
Sequence-level prediction improves both action scoring and localization accuracy compared with operating at the frame level.
Takeaways & Limitations
Anchor cuboids assume fixed spatial extent over time, although experiments show they handle moving actors for short sequences.
Abstract
from arXiv · showhide
Current state-of-the-art approaches for spatio-temporal action localization rely on detections at the frame level that are then linked or tracked across time. In this paper, we leverage the temporal continuity of videos instead of operating at the frame level. We propose the ACtion Tubelet detector (ACT-detector) that takes as input a sequence of frames and outputs tubelets, i.e., sequences of bounding boxes with associated scores. The same way state-of-the-art object detectors rely on anchor boxes, our ACT-detector is based on anchor cuboids. We build upon the SSD framework. Convolutional features are extracted for each frame, while scores and regressions are based on the temporal stacking of these features, thus exploiting information from a sequence. Our experimental results show that leveraging sequences of frames significantly improves detection performance over using individual frames. The gain of our tubelet detector can be explained by both more accurate scores and more precise localization. Our ACT-detector outperforms the state-of-the-art methods for frame-mAP and video-mAP on the J-HMDB and UCF-101 datasets, in particular at high overlap thresholds.
1. Introduction
Action localization methods commonly process frames independently and link detections afterward, but single-frame evidence can be ambiguous. ACT-detector instead uses frame sequences, anchor cuboids, and temporally aggregated features to produce tubelets.
- Action localization must classify actions and localize them in space and time despite variability, clutter, low quality, occlusion, and viewpoint changes.
- Frame-level methods link or track independently generated detections, so they do not exploit temporal continuity.
- ACT-detector takes a fixed-length frame sequence and outputs tubelets, or temporally ordered bounding boxes with associated scores.
- The detector extends SSD by scoring and regressing densely sampled anchor cuboids using convolutional features from all frames in the sequence.
- The overview extracts shared-weight convolutional features, stacks subsequent-frame features, and predicts anchor-cuboid scores and coordinates that form tubelets.
- Sequence input improves action scoring and localization accuracy, while ACT-detector achieves state-of-the-art frame-mAP and video-mAP on J-HMDB and UCF-101, especially at high overlap thresholds.
2. Related work
CNN-based action localization evolved from proposals and actionness methods toward frame-level object detectors whose detections are linked across time. ACT-detector extends anchor-box detection to anchor cuboids to exploit temporal structure directly.
- CNN object detectors such as R-CNN, Faster R-CNN, and SSD underpin most recent action-localization approaches.
- Modern detectors classify and regress densely sampled anchor boxes, while SSD selects feature layers according to box size.
- ACT-detector extends anchor-box detectors to anchor cuboids, yielding significant improvement for action localization.
- Earlier action-localization methods used sliding windows with fixed cuboid assumptions or video proposals based on super-voxels and dense trajectories.
- More recent approaches commonly detect human action classes at the frame level and then link detections across time.
- Frame-level systems include two-stream R-CNN variants, proposal-based methods with multi-frame optical flow, and real-time SSD with online linking.
3. ACtion Tubelet (ACT) detector
The ACT-detector extends SSD to process short frame sequences and predict spatio-temporal tubelets using anchor cuboids. It combines jointly regressed tubelets with two-stream RGB and flow inputs, then links them into action tubes.
- ACT-detector: The ACT-detector takes K frames as input and outputs tubelets containing a sequence of bounding boxes with one confidence score per action class.The approach applies the tubelet extension on top of SSD.
- ACT-detector: Convolutional features are computed independently for each frame with shared weights, then temporally stacked for action scoring and anchor-cuboid regression.The stacked features feed separate layers for classification and cuboid regression.
- ACT-detector: Anchor cuboids assume fixed spatial extent across K frames, while regression predicts separate box coordinates per frame and can deform the resulting tubelet over time.The receptive field also provides context beyond the cuboid, supporting localization when actors move outside it.
- ACT-detector: Training assigns anchor cuboids to ground-truth tubelets and combines confidence and regression losses over positive assignments.The confidence loss uses softmax classification, while the regression loss uses Smooth-L1 offsets for box centers, widths, and heights averaged over K frames.
- Two-stream ACT-detector: The two-stream detector processes RGB sequences and flow images, combines their outputs using union or late fusion, and retains the RGB-regressed tubelet for fusion.Late fusion outperforms union fusion in the reported experiments.
- From action tubelets to spatio-temporal tubes: Overlapping tubelets are linked online into action tubes, with shared convolutional features making computation efficient for consecutive K-frame sequences.The method retains the 10 highest-scored tubelets per class after NMS and uses a temporal overlap threshold τ = 0.2 for linking.
4. Experimental results
Experiments across three datasets show that ACT-detector benefits from temporal sequences, with improved modality fusion, tubelet lengths, error profiles, and state-of-the-art performance.
- 4.2. Validation of anchor cuboids: Anchor-cuboid recall at IoU=0.5 remains ⩾98% up to K = 6 and over 95% for K =10 across all datasets.Recall starts dropping significantly for much longer sequences such as K = 32.
- 4.3. Tubelet modality: RGB outperforms flow for every dataset and K, while combining modalities improves detection performance.Late fusion consistently beats union fusion by 1% to 4% frame-mAP and is used thereafter.
- 4.4. Tubelet length: Increasing tubelet length yields up to 10% frame-mAP gains by K = 6 or 8 versus per-frame detection, but performance drops for K > 8 on J-HMDB and UCF-101.The paper selects K = 6 because it provides good overall results.
- 4.4. Tubelet length: For fast actors, K = 6 improves frame-mAP over K = 1 by +8% on UCF-Sports, +9% on J-HMDB, and +3% on UCF-101.The result supports the ability of tubelets to handle large actor displacements.
- 4.5. Error breakdown analysis: With K = 6, missed detections decrease significantly and classification errors decrease, while localization and other errors change little.On J-HMDB, missed detections are reduced by a factor of 2 compared with K = 1.
- 4.6. Comparison to the state of the art: With K = 6, ACT-detector outperforms state-of-the-art methods in frame-mAP on UCF-Sports, J-HMDB, and UCF-101, while running at 25-30fps.Its performance advantage is especially pronounced at high overlap thresholds, and its runtime matches frame-based SSD.
5. Conclusions
The ACT-detector leverages temporal continuity by processing frame sequences and producing tubelets rather than single-frame detections. Built on SSD with anchor cuboids, it improves classification and localization and achieves state-of-the-art results, especially at high overlap thresholds.
- The ACT-detector processes sequences of frames and outputs tubelets instead of operating on individual frames.
- Built upon SSD, the method scores and regresses anchor cuboids across frame sequences.
- The detector improves both classification and localization and achieves state-of-the-art results, particularly at high overlap thresholds.