Source-linked AI summary
TubeR: Tubelet Transformer for Video Action Detection
Jiaojiao Zhao, Yanyi Zhang, Xinyu Li, Hao Chen, Shuai Bing, Mingze Xu, Chunhui Liu, Kaustav Kundu, Yuanjun Xiong, Davide Modolo, Ivan Marsic, Cees G. M. Snoek, Joseph Tighe
TL;DR
Spatio-temporal action detection requires localizing people, linking them into action tubes, and assigning action labels, while fixed cuboids can limit flexible tubelet representations. TubeR uses learned tubelet queries and contextual classification to detect actions, and reports state-of-the-art performance with better efficiency than previous works. Its 3D backbone remains a constraint for very long videos.
Problem
Spatio-temporal action detection combines per-frame person localization, linking detections into action tubes, and predicting action classes, while fixed cuboids limit flexible long-range tubelet representation.
Source: §1 · p. 1Method
Results
TubeR achieves state-of-the-art performance and better efficiency than previous works.
Source: §5 · p. 8Takeaways & Limitations
TubeR generates linked-box tubelets with action predictions without positional hypotheses and scales well to longer video clips.
Source: §5 · p. 8The 3D backbone uses 90% of computation and 67% of memory, restricting TubeR on long videos.
Source: §5 · p. 8
Abstract
from arXiv · showhide
We propose TubeR: a simple solution for spatio-temporal video action detection. Different from existing methods that depend on either an off-line actor detector or hand-designed actor-positional hypotheses like proposals or anchors, we propose to directly detect an action tubelet in a video by simultaneously performing action localization and recognition from a single representation. TubeR learns a set of tubelet-queries and utilizes a tubelet-attention module to model the dynamic spatio-temporal nature of a video clip, which effectively reinforces the model capacity compared to using actor-positional hypotheses in the spatio-temporal space. For videos containing transitional states or scene changes, we propose a context aware classification head to utilize short-term and long-term context to strengthen action classification, and an action switch regression head for detecting the precise temporal action extent. TubeR directly produces action tubelets with variable lengths and even maintains good results for long video clips. TubeR outperforms the previous state-of-the-art on commonly used action detection datasets AVA, UCF101-24 and JHMDB51-21.
1. Introduction
TubeR targets spatio-temporal action detection with flexible tubelets that can vary in location and scale, rather than fixed cuboids. It combines tubelet queries with contextual classification and reports state-of-the-art results on three action-detection datasets.
1. Introduction
- Unlike short 3D cuboids, TubeR generates flexible tubelets that change location and size over time, aggregating visual information across longer periods.This formulation jointly localizes and recognizes action tubelets.Source: §1 · p. 1
- TubeR uses decoder queries to represent people and their actions across video sequences without restricting tubelets to fixed cuboids.Source: §1 · p. 2
- Its tubelet-query formulation supports arbitrary location and scale, while the context-aware classification head aggregates short- and long-term context.Source: §1 · p. 2
- TubeR reports state-of-the-art results on three challenging action-detection datasets.The introduction does not name the datasets or provide metric values here.Source: §1 · p. 2
2. Related Work
Prior work detects actions frame by frame or through tubelet-level hypotheses, while transformer-based approaches have also used proposals for localization. TubeR instead targets unified tubelet-level localization and recognition.
2. Related Work
- Frame-level methods localize and recognize actors per frame, then link predictions into action tubes, often using anchors or an offline person detector.Other approaches add optical flow or 3D convolution for temporal information, and some model actor-object relations.Source: §2 · p. 2
- Tubelet-level methods use anchors, cuboids, or center-position hypotheses, which can make long-video processing difficult; TubeR learns queries to represent dynamic tubelets.The passage also describes TubeR as reformulating detection as sequence-to-sequence learning and modeling within-tubelet temporal correlations.Source: §2 · p. 2
- A prior video action transformer uses a region-proposal network for localization and aggregates spatio-temporal context for recognition; TubeR unifies localization and recognition.Source: §2 · pp. 2–3
3. Action Detection by TubeR
TubeR transforms video features into tubelet-specific representations, then predicts tubelet coordinates and action labels. Its encoder, decoder, task-specific heads, and losses support tubelets that can be sparse or dense over time.
3. Action Detection by TubeR
- TubeR jointly predicts tubelet coordinates and action labels from tubelet-specific features, with temporal down-sampling producing sparse tubelets for long clips and no down-sampling producing dense tubelets for short clips.The transformer maps backbone features to N tubelet features across Tout output frames; the coordinate and classification heads operate on those features.Source: §3 · p. 3
3.1. TubeR Encoder
- The encoder applies spatio-temporal self-attention to backbone features, with optional temporal down-sampling to reduce the input sequence length and memory requirement.Each encoder layer also includes normalization layers and a feed-forward network.Source: §3.1 · p. 3
3.2. TubeR Decoder
- Tubelet queries use spatial self-attention to model within-frame relations and temporal self-attention to model within-tubelet relations across time; cross-attention then decodes tubelet-specific features.The decoder's output is sparse when temporal pooling makes Tout < Tin and dense when Tout = Tin.Source: §3.2 · pp. 3–4
3.3. Task-Specific Heads
- Task-specific heads classify tubelets using short-term backbone context or compressed long-term context, while action-switch scores identify boxes within a tubelet that depict the action.The action-switch head is intended to provide a more precise temporal extent; coordinates are regressed across Tout frames.Source: §3.3 · p. 4
3.4. Losses
- Training combines action-switch, classification, per-frame box-matching, and box-IoU losses, using down-sampled coordinate ground truth when tubelets are sparse.The matching uses Hungarian matching, and the loss weights are λ1=1, λ2=5, λ3=2, and λ4=2.Source: §3.4 · pp. 4–5
4. Experiments
Experiments evaluate TubeR through component ablations, comparisons on AVA, UCF101-24, and JHMDB51-21, and visualizations across challenging video scenarios. TubeR reports gains over prior methods and handles varied temporal and spatial conditions in the visualized cases.
4.1. Experimental Setup
- Evaluation reports video-mAP at multiple IoUs for UCF101-24 and JHMDB51-21, and frame-mAP@IoU=0.5 for AVA's keyframe annotations.Experiments use UCF101-24 split 1, average results across all three JHMDB51-21 splits, and evaluate both AVA v2.1 and v2.2.Source: §4.1 · p. 5
4.2. Ablations
- Action switch raises UCF101-24 video-mAP from 53.8% to 57.7%; context adds +4.3% on AVA, with about 15s of long-term context best.The context head does not significantly improve UCF101-24; overwhelming long-term context can harm performance.Source: §4.2 · pp. 5–6
- Tubelet query sets add +4.1% video mAP on UCF101-24 versus a single embedding; tubelet attention adds 0.9% there and 0.3% on AVA.The ablations report video-mAP for UCF101-24 and frame-mAP for AVA.Source: §4.2 · p. 5, Table 1 · p. 6
4.3. Frame-Level State-of-the-Art
- On AVA 2.1, TubeR beats WOO by 0.9% and VTr by 1.2%; against a two-stage model with the same CSN backbone, it gains +4.4%.The AVA 2.1 comparisons use comparable backbones and the same inference protocol.Source: §4.3 · p. 6, Table 2 · p. 7
- On AVA 2.2, TubeR exceeds WOO by +5.1% and MViT by +4.7% under single-view protocol, and reaches 83.2 frame-mAP on UCF101-24 with CSN-152.The AVA comparisons use the same single-view protocol; the UCF101-24 result is reported for frame-level evaluation.Source: §4.3 · pp. 6–7, Table 3 · p. 7
4.4. Video-Level State-of-the-Art
- +4.4% video-mAP@IoU=0.5 over [26] on UCF101-24 with a 2D backbone and no optical flow; direct tubelet generation runs at 156 fps versus CFAD's 130 fps.The speed test follows CFAD's UCF101-24 inference protocol.Source: Table 4 · p. 7, §4.4 · pp. 7–8
4.5. Visualization
- Visualizations show separated action-specific tubelets and action-switch start/stop timing, plus performance in shot changes, distant movement, and small-actor scenes.The examples also include multiple concurrent actions and scale changes.Source: §4.5 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8
5. Discussion and Conclusion
TubeR produces linked bounding-box tubelets with action predictions without positional hypotheses, achieving state-of-the-art performance and better efficiency than previous works. Its 3D backbone and the query requirements of one-pass long-video processing constrain its use on very long videos.
Limitations
- The 3D backbone uses 90% of computation (FLOPs) and 67% of memory, restricting TubeR's application to long videos.The authors propose exploring transformer-based video embeddings in future work.Source: §5 · p. 8
- One-pass long-video processing would likely require many queries to cover per-person actions, causing memory issues in self-attention layers.The authors suggest person tubelets as a possible solution, requiring one query per person instance rather than splitting tubelets when actions change.Source: §5 · p. 8
Potential negative impact
- The authors identify potential unintended uses of action detection and advocate responsible use and compliance with applicable laws and regulations.Source: §5 · p. 8
Conclusion
- TubeR achieves state-of-the-art performance and better efficiency than previous works while producing action-predicted tubelets without positional hypotheses.The conclusion describes each tubelet as a set of linked bounding boxes.Source: §5 · p. 8