PaperlayerView exact revision on arXiv

Source-linked AI summary

TubeDETR: Spatio-Temporal Video Grounding with Transformers

Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, Cordelia Schmid

arXiv:2203.16434v2cs.CVcs.CLcs.LG

TL;DR

Spatio-temporal video grounding asks a model to localize a text-described object across space and time, requiring efficient joint modeling of video and language. TubeDETR combines a sparse-interaction video-text encoder with a space-time decoder, and outperforms state-of-the-art methods on VidSTG and HC-STVG.

  • Problem

    Spatio-temporal video grounding must localize a text-described object across space and time, despite videos containing similar-looking or similarly acting entities.

    Source: §1 · p. 1

  • Method

    TubeDETR uses a sparse-frame video-text encoder with a fast visual branch and a space-time decoder to jointly model interactions and predict grounded tubes.

    Source: §1 · pp. 1–2, §3.1 · p. 3, §3.2 · p. 3

  • Results

    TubeDETR outperforms state-of-the-art methods on the VidSTG and HC-STVG benchmarks.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The results support TubeDETR’s effectiveness for spatio-temporal video grounding on VidSTG and HC-STVG.

    Source: §1 · p. 2, §5 · p. 8

Abstract

from arXiv · show

We consider the problem of localizing a spatio-temporal tube in a video corresponding to a given text query. This is a challenging task that requires the joint and efficient modeling of temporal, spatial and multi-modal interactions. To address this task, we propose TubeDETR, a transformer-based architecture inspired by the recent success of such models for text-conditioned object detection. Our model notably includes: (i) an efficient video and text encoder that models spatial multi-modal interactions over sparsely sampled frames and (ii) a space-time decoder that jointly performs spatio-temporal localization. We demonstrate the advantage of our proposed components through an extensive ablation study. We also evaluate our full approach on the spatio-temporal video grounding task and demonstrate improvements over the state of the art on the challenging VidSTG and HC-STVG benchmarks. Code and trained models are publicly available at https://antoyang.github.io/tubedetr.html.

1. Introduction

TubeDETR addresses spatio-temporal video grounding, which requires linking language to spatial regions and temporal boundaries. Its transformer architecture jointly models temporal, spatial, and multimodal interactions and outperforms prior methods on VidSTG and HC-STVG.

1. Introduction

  • Grounding language in visual content is presented as important for building powerful, explainable vision-and-language models and analyzing their spatial-temporal associations.This requires more than associating global visual and textual representations [57] [62].Source: §1 · p. 1
  • Spatio-temporal video grounding localizes a tube of bounding boxes for a described object, amid videos where entities may look or act similarly.The task combines visual grounding and temporal localization [33] [59] [74] [25] [30].Source: §1 · p. 1
  • TubeDETR outperforms state-of-the-art methods on VidSTG and HC-STVG with a transformer architecture for joint spatio-temporal grounding.The model uses a dual-stream encoder and a space-time decoder.Source: §1 · p. 2
  • Its dual-stream encoder and space-time decoder jointly model temporal, spatial, and multimodal interactions for grounding.The encoder uses sparse-frame multimodal interactions and a lightweight fast branch; the decoder combines temporal self-attention with time-aligned cross-attention.Source: §1 · pp. 1–2

2. Related Work

Related work spans visual grounding, temporal localization, video modeling, and vision-language transformers. TubeDETR extends a one-stage grounding framework to video while preserving spatial detail and modeling temporal interactions.

Spatio-temporal video grounding

  • TubeDETR extends MDETR’s one-stage framework to spatio-temporal video grounding with temporal localization losses, slow-fast encoding, and space-time decoding.MDETR [37] is among one-stage approaches that do not rely on pre-extracted object proposals.Source: §2 · p. 2
  • Temporal-localization methods reason about time but do not preserve spatial information, while spatio-temporal grounding combines temporal and visual grounding.Some approaches rely on pre-extracted tube or object proposals [15] [72] [84] [102]; TubeDETR does not require them, unlike that proposal-based setup.Source: §2 · p. 2
  • STVGBert uses deconvolutions and symmetrically models temporal and spatial interactions, whereas TubeDETR grounds with a transformer decoder and reasons separately over those dimensions.STVGBert extends VilBERT [54], pretrained on Conceptual Captions.Source: §2 · p. 2

Temporal modeling for video understanding

  • Unlike moment-retrieval work that does not preserve spatial information, TubeDETR modifies a pretrained encoder and decoder to reason about language as well as video.Its dual-branch encoder is related to SlowFast networks [23] [82], but both streams use features from the same backbone and address multimodal-computation complexity.Source: §2 · p. 2

Vision and language

  • Many video-text transformers use object or pooled features that lose spatial detail; TubeDETR instead targets detailed frame-level representations for spatial and temporal localization.Other video-text transformers preserve spatial information but typically learn global representations for video-level prediction [2] [5] [42] [94].Source: §2 · p. 2

3. Method

TubeDETR encodes spatial and multimodal interactions over short video clips, then decodes features across the video to predict a grounded spatio-temporal tube. The method also specifies its training losses and pretrained weight initialization.

3.1. Overview

  • Given a video and language query, TubeDETR predicts a tube of boxes with temporal boundaries, using short-clip interaction modeling and a full-video decoder for efficiency.The video may contain hundreds of frames represented by tens of thousands of spatio-temporal features.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2. Video-Text Encoder

  • The video-text encoder computes spatial and multimodal interactions on every k-th frame, while a lightweight fast visual branch preserves frame-rate features to recover sampled-out detail.The branches are fused into per-frame video-text features.Source: §3.2 · pp. 3–4, Figure 3 · p. 4

3.3. Space-Time Decoder

  • The decoder uses temporal self-attention across frames and time-aligned cross-attention, then predicts per-frame boxes and start/end probabilities for the tube.Each time query cross-attends only to video-text features from its corresponding frame.Source: §3.3 · pp. 4–5, Figure 4 · p. 5

3.4. Training loss

  • Training combines box-coordinate L1 and generalized-IoU losses with KL divergence for start/end distributions and guided temporal-attention loss.Target start and end distributions are quantized Gaussians centered on annotated times, with standard deviation 1.Source: §3.4 · p. 5

3.5. Weight initialization

  • TubeDETR initializes its video-text encoder and multi-frame decoder from corresponding MDETR weights, excluding the fast and aggregation modules and temporal localization head.The MDETR weights were pretrained on Flickr30k, MS COCO, and Visual Genome.Source: §3.5 · p. 5

4. Experiments

TubeDETR is evaluated on VidSTG and HC-STVG, with ablations examining its decoder, initialization, and video-text encoder. The full model outperforms prior methods on both benchmarks, while qualitative examples include both successful tubes and a confusion between similar objects.

4.1. Experimental setup

  • Evaluation covers VidSTG and HC-STVG, using m vIoU and vIoU@R for spatio-temporal grounding, m tIoU for temporal localization, and m sIoU for spatial grounding.Videos are sampled at 5 frames per second, capped at 200 sampled frames; the HC-STVG2.0 test set was not publicly available at the time of writing.Source: §4.1 · p. 6

4.2. Ablation studies

  • Adding time encoding and temporal self-attention improves vIoU@0.3 by 17.9% over the space-only decoder, mainly through temporal localization (+22.0% m tIoU).Time encoding contributes +13.4% vIoU@0.3, with temporal self-attention adding a further +4.5%; spatial grounding rises by +0.7% m sIoU.Source: §4.2 · p. 6, Table 1 · p. 6
  • MDETR pretraining improves vIoU@0.3 by 8.1%, and transferring its spatial self-attention weights to temporal self-attention adds a further 3.3%.The reported initialization ablation also finds a +9.2% gain in m sIoU for spatial grounding.Source: §4.2 · p. 6, Table 2 · p. 6, Table 4 · p. 7
  • Higher resolution improves grounding but increases memory use; the best results across both datasets use temporal stride k = 4 and resolution 352.At 224 resolution, increasing temporal stride to 2 or 5 reduces peak memory by 7.7GB or 12.1GB, respectively.Source: Table 4 · p. 7, Table 3 · p. 7
  • The fast branch improves vIoU@0.3 by 2.1% on VidSTG and 2.9% on HC-STVG2.0 with low memory overhead, recovering details lost through slow-branch temporal sampling.The comparison is against a slow-only variant with f = 0 and g = 0.Source: §4.2 · p. 7, Table 3 · p. 7

4.3. Comparison to the state of the art

  • With MDETR initialization, TubeDETR outperforms prior methods by a large margin on VidSTG and HC-STVG; ImageNet-only initialization is competitive despite using fewer annotations.The comparison uses the space-time decoder with time encoding and temporal self-attention, temporal stride k = 4, and resolution 352.Source: §4.3 · p. 7, Table 4 · p. 7

4.4. Qualitative examples

  • VidSTG examples show accurate tubes for a cat biting an adult, a man grabbing a small ball, and skis under a skier, but the model confuses a background ball with a balloon.The paper notes that the ball and balloon are visually and semantically similar.Source: §4.4 · p. 8, Figure 5 · p. 8

5. Conclusion

TubeDETR combines a video-text encoder with a space-time transformer decoder for spatio-temporal video grounding. The paper reports effectiveness of the decoder, a performance-memory trade-off from the encoder, and state-of-the-art results on two benchmarks.

5. Conclusion

  • TubeDETR combines a space-time transformer decoder with a video-text encoder that efficiently encodes spatial and multimodal interactions.The conclusion reports that the approach outperforms state-of-the-art methods on VidSTG and HC-STVG.Source: §5 · p. 8
  • The authors identify extending the decoder to multiple objects per frame or multiple events per video as future work.They also point to more efficient alternatives to self-attention as a promising direction.Source: §5 · p. 8
  • The conclusion reports the decoder's effectiveness and the encoder's performance-memory trade-off as demonstrated contributions.Source: §5 · p. 8

Appendix

The appendix gathers supplementary visualizations, implementation information, and additional ablation results, and includes a broader-impact discussion.

Appendix

  • The appendix provides attention visualizations, additional implementation details, and VidSTG ablations split by sentence type.It also contains an ablation of the fast and aggregation modules and a broader-impact discussion.Source: Appendix · p. 13

A. Visualization of space, time and language attention patterns in the decoder

The appendix visualizes the decoder's attention across space, language, and time. The example shows changing word attention over time and long-range interactions among time queries.

A. Visualization of space, time and language attention patterns in the decoder

  • Visual attention focuses on people receiving the sports ball and gesturing, while the words “adult” and “grabs” receive the most overall textual attention.Source: A. Visualization of space, time and… · p. 13
  • Attention weights on words such as “sports” and “ball” vary over time, and time queries attend to temporally distant queries around frames eight through sixteen.The figure also marks the predicted start and end times of the output tube.Source: A. Visualization of space, time and… · p. 13

B. Additional implementation details

The implementation uses a standard transformer and optimization setup, while Figure 6 visualizes time-aligned textual cross-attention weights.

B. Additional implementation details

  • The transformer uses 8 heads and feed-forward layers with hidden dimension 2048.Dropout is 0.1 in transformer layers and 0.5 in the temporal localization head.Source: B. Additional implementation details · p. 13
  • AdamW uses weight decay 1e−4, with initial learning rates of 1e−5 for the visual backbone and 5e−5 for the rest of the network.The text encoder uses linear warm-up and scheduling; the rest of the network uses a constant learning rate.Source: B. Additional implementation details · p. 13
  • Figure 6 shows time-query attention to time-aligned, visually contextualized text features; lighter colors indicate higher weights.Weights are averaged across 8 heads and 6 layers, then normalized by each row’s maximum for visualization.Source: Figure 6 · p. 13

C. Detailed ablation results

The ablations examine decoder timing, initialization, sampling and resolution, and encoder-branch design; attention visualizations show where decoder queries attend.

Attention visualizations

  • Figure 7 shows visual-modality attention focused on humans receiving the sports ball and gesturing; Figure 8 maps time-query self-attention across query times.Figure 7 overlays predicted and ground-truth tubes; Figure 8 marks the predicted start and end times.Source: Figure 7 · p. 14, Figure 8 · p. 15

Space-time decoder

  • +18.3% on vIoU@0.3 for declarative sentences and +17.5% on vIoU@0.3 for interrogative sentences when adding time encoding and temporal self-attention to the space-only decoder.The gains are mostly temporal localization: +22.0% on m_tIoU for declarative and +22.1% on m_tIoU for interrogative sentences; spatial grounding increases +0.9% and +0.5% on m_sIoU, respectively.Source: Table 5 · p. 15, C. Detailed ablation results · p. 16

Initialization

  • +9.4% on vIoU@0.3 for declarative sentences and +6.9% on vIoU@0.3 for interrogative sentences with pretraining.Pretraining gains mainly come from spatial grounding (+11.7% on m_sIoU for declarative and +7.1% on m_sIoU for interrogative sentences); MDETR decoder weights add +3.2% and +3.3% on vIoU@0.3, respectively.Source: C. Detailed ablation results · p. 16, Table 6 · p. 16

Impact of spatial resolution and temporal stride

  • Temporal stride k = 4 with resolution 352 gives the best m_vIoU and vIoU@R results, while higher resolution improves performance and memory trade-offs against the k = 1 baseline.The encoder enables training at higher resolutions for a given memory usage.Source: C. Detailed ablation results · p. 16, Table 7 · p. 16

Impact of the fast branch

  • +2.2% vIoU@0.3 for declarative sentences and +1.9% vIoU@0.3 for interrogative sentences with the fast branch over the slow-only variant, at low memory overhead.The authors attribute this to recovery of spatio-temporal details lost through slow-branch temporal sampling.Source: C. Detailed ablation results · p. 16, Table 7 · p. 16

D. Additional Experiments

Additional encoder ablations compare branch and module designs, and the paper notes potential environmental relevance from examining training memory use.

Additional experiments

  • The work also examines memory use when learning these models and could help promote lighter models with reduced environmental impact.The paper frames this as a potential contribution.Source: D. Additional Experiments · p. 17

Design of the fast and aggregation modules

  • The slow-fast encoder outperforms a fast-only variant, supporting the importance of the slow multi-modal branch.The fast-only variant concatenates fast visual-only features with text features without the slow branch.Source: D. Additional Experiments · p. 17
  • Preserving spatial information in each frame outperforms spatial pooling in the fast branch; the proposed design also slightly improves over gating and a fast temporal transformer.The comparisons suggest that forcing use of the slow branch is not helpful and that additional temporal interaction modeling in the encoder is not necessarily helpful.Source: D. Additional Experiments · p. 17

E. Broader Impact

The authors state that the work’s potential positive or negative impacts depend on its application, including possible use in video surveillance.

E. Broader Impact

  • The work’s potential positive or negative impacts depend on the application.The authors give video surveillance as a possible use that could lead to questionable use.Source: E. Broader Impact · p. 17
  • Video surveillance is identified as a possible application that may lead to questionable use.The authors do not claim that this outcome is inevitable.Source: E. Broader Impact · p. 17

Related papers

Loading 2203.16434v2…