PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences

Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, Lianli Gao

arXiv:2001.06891v3cs.CV

TL;DR

STVG addresses localization of queried-object tubes in untrimmed videos for both explicit-object and unknown-object sentences. STGRN uses spatio-temporal graph reasoning and direct tube localization, while VidSTG provides a benchmark; experiments report effectiveness across the evaluated settings.

  • Problem

    STVG requires grounding short-lived objects in untrimmed videos and resolving interrogative sentences whose target objects are unknown from explicit class or appearance cues.

  • Method

    STGRN builds a spatio-temporal region graph, performs cross-modal graph reasoning, and directly retrieves tubes without tube pre-generation.

  • Results

    STGRN outperforms all baselines on declarative and interrogative sentences with or without temporal ground truth.

  • Takeaways & Limitations

    VidSTG supplies a large-scale benchmark for STVG, and the experiments support STGRN's effectiveness for the task.

Abstract

from arXiv · show

In this paper, we consider a novel task, Spatio-Temporal Video Grounding for Multi-Form Sentences (STVG). Given an untrimmed video and a declarative/interrogative sentence depicting an object, STVG aims to localize the spatio-temporal tube of the queried object. STVG has two challenging settings: (1) We need to localize spatio-temporal object tubes from untrimmed videos, where the object may only exist in a very small segment of the video; (2) We deal with multi-form sentences, including the declarative sentences with explicit objects and interrogative sentences with unknown objects. Existing methods cannot tackle the STVG task due to the ineffective tube pre-generation and the lack of object relationship modeling. Thus, we then propose a novel Spatio-Temporal Graph Reasoning Network (STGRN) for this task. First, we build a spatio-temporal region graph to capture the region relationships with temporal object dynamics, which involves the implicit and explicit spatial subgraphs in each frame and the temporal dynamic subgraph across frames. We then incorporate textual clues into the graph and develop the multi-step cross-modal graph reasoning. Next, we introduce a spatio-temporal localizer with a dynamic selection method to directly retrieve the spatio-temporal tubes without tube pre-generation. Moreover, we contribute a large-scale video grounding dataset VidSTG based on video relation dataset VidOR. The extensive experiments demonstrate the effectiveness of our method.

1. Introduction

The paper introduces STVG, which localizes queried-object tubes in untrimmed videos from declarative or interrogative sentences. It addresses short-lived objects, unknown interrogative targets, weak tube pre-generation, and limited relationship modeling with STGRN and the VidSTG benchmark.

  • STVG localizes the queried object's spatio-temporal tube from an untrimmed video and a declarative or interrogative sentence.
  • Objects may appear only briefly, making temporal-boundary determination difficult when sentences describe short-term object states.
  • Interrogative sentences identify unknown objects through their actions and spatial relationships with other objects rather than explicit class or appearance cues.
  • STGRN builds a spatio-temporal region graph, performs cross-modal reasoning, and directly localizes tubes without tube pre-generation.
  • VidSTG adds multi-form sentence annotations to VidOR as a large-scale benchmark for STVG.
  • Extensive experiments report that STGRN is effective for the proposed task.

2. Related Work

Prior work covers temporal grounding, visual grounding, and video grounding, but the cited approaches use different representations and settings. STGRN instead combines spatio-temporal graph reasoning with direct tube retrieval.

  • Temporal grounding detects video clips matching sentences, while visual grounding localizes described objects through visual-language matching.
  • Recent visual-grounding methods decompose expressions, use co-attention, or model relationships between objects.
  • STGRN's architecture applies a spatio-temporal graph encoder for multi-step cross-modal reasoning before localizing tubes from region-level features.

3. The Proposed Method

STGRN builds a spatio-temporal region graph, injects textual clues through multi-step cross-modal reasoning, and directly localizes queried object tubes without pre-generating tube candidates.

  • Spatio-Temporal Region Graph: STGRN parses each video into region nodes connected by implicit spatial, explicit spatial, and temporal dynamic subgraphs.The spatial graphs model within-frame relationships, while the temporal graph connects regions across nearby frames to capture object dynamics.
  • Spatio-Temporal Region Graph: The implicit spatial graph fully connects regions within each frame, whereas the explicit graph represents directed, labeled relationships between region pairs.Explicit edges use predicted predicates and include three directions plus 51 edge-label types.
  • Spatio-Temporal Region Graph: Temporal edges connect regions across adjacent frames in forward, backward, and self-loop directions to model object transformations.The construction limits temporal connections to nearby frames and selects regions using linking scores based on appearance and spatial overlap.
  • Multi-Step Cross-Modal Graph Reasoning: Cross-modal fusion injects sentence information into region features by attending over words and using a textual gate to weaken text-irrelevant regions.The resulting features combine filtered visual regions with region-aware textual representations before graph reasoning.
  • Multi-Step Cross-Modal Graph Reasoning: Multi-step spatio-temporal graph convolution combines implicit, explicit, and temporal reasoning to learn relation-aware region features.Spatial convolution captures within-frame visual relationships, while temporal convolution models dynamics across frames; repeated layers support multi-order relation modeling.

4. Dataset

VidSTG is a large-scale spatio-temporal grounding dataset created by augmenting VidOR with declarative and interrogative descriptions of queried objects and their relationships.

  • Dataset Construction: VidSTG augments VidOR with sentence annotations to provide a benchmark for spatio-temporal video grounding.VidOR supplies object boxes, relations, temporal boundaries, and spatio-temporal tubes, reducing the need for new bounding-box annotation.
  • Dataset Construction: VidOR contains 10,000 videos, 80 object categories, and 50 relation-predicate categories with dense object and relation annotations.The relation categories include 8 spatial relations and 42 action relations.
  • Dataset Annotation: The dataset describes selected subjects or objects with appearance, relationships, and visual environments, while interrogative descriptions omit queried-object appearance.Interrogative sentences therefore rely on relationships with other objects rather than explicit appearance or class descriptions.
  • Dataset Statistics: VidSTG contains 99,943 sentence descriptions for 44,808 video-triplet pairs and 79 queried-object types.Average video duration is 28.01s, average object-tube duration is 9.68s, and declarative and interrogative sentences average 11.12 and 8.98 words.

5. Experiments

Experiments evaluate STGRN on VidSTG using grounding metrics, baselines, ablations, and qualitative examples. Results show benefits from temporal dynamics, graph reasoning, dynamic tube selection, and appropriate graph depth, while interrogative sentences remain harder.

  • Experimental settings: STGRN is evaluated on VidSTG with m tIoU, m vIoU, and vIoU@R, alongside extended GroundeR, STPR, and WSSTG baselines.The baselines combine temporal localization methods with frame- or tube-level grounding approaches.
  • Overall results: GroundeR-based framewise grounding performs worse than STPR- and WSSTG-based methods, supporting the importance of modeling temporal object dynamics across frames.The comparison concerns methods that independently ground sentences in each frame versus tube-level approaches.
  • Overall results: Interrogative sentences perform worse than declarative sentences because their unknown objects must be grounded through relationships with other objects.This setting lacks explicit object characteristics such as class and visual appearance.
  • Overall results: STGRN outperforms all baselines for declarative and interrogative sentences, with and without temporal ground truth.The result is attributed to cross-modal spatio-temporal graph reasoning and direct spatio-temporal localization.
  • Tube generation: Dynamic selection outperforms greedy selection in STGRN, indicating that trajectory smoothness improves spatio-temporal tube quality.Greedy selection chooses regions with the highest matching scores, whereas dynamic selection is introduced to reduce excessive displacement between adjacent boxes.
  • Ablation study: The full spatio-temporal graph outperforms ablations; among single subgraphs Gexp performs best, while Gexp plus Gtem is strongest among two-subgraph variants.These results support the usefulness of implicit spatial, explicit spatial, and temporal dynamic relations, with explicit and temporal modeling especially important.
  • Ablation study: STGRN performs best with two graph-convolution layers, whereas one layer undercaptures relationships and excessive depth can cause region over-smoothing.The layer-depth trend is consistent across m tIoU, m vIoU, and sentence types.
  • Qualitative analysis: A qualitative example shows STGRN producing a more precise temporal segment and more reasonable tube than WSSTG+L-net for short-term parallel actions.The example involves a child grabbing a man’s hands and jumping off a blue slide.

6. Conclusion

The paper introduces the STVG task and VidSTG dataset, and proposes STGRN to directly localize spatio-temporal tubes through region-level relationship modeling.

  • STVG and VidSTG are introduced as a new spatio-temporal video grounding task and large-scale dataset.
  • STGRN captures region relationships with temporal object dynamics and directly localizes spatio-temporal tubes from regions.

7. Dataset Details

VidSTG is constructed from VidOR by annotating queried objects with declarative and interrogative descriptions, yielding broad object coverage and both temporal and spatio-temporal grounding annotations.

  • VidSTG contains 6,924 videos and 99,943 sentences covering 44,808 video-triplet pairs.The dataset is split into 5,563 training, 618 validation, and 743 testing videos.
  • Annotations describe queried subjects or objects through their appearance, relationships, and visual environments.A video-triplet pair may have multiple sentences, including declarative and interrogative forms.
  • VidSTG describes 79 object types: 3 persons, 27 animals, and 49 other objects.One original VidOR object type, stingray, is absent after cleaning and annotation.
  • Unlike prior datasets, VidSTG provides temporal clips and spatio-temporal tubes, richer object variety, more sentence descriptions, and multi-form sentences.Prior temporal grounding datasets lack spatio-temporal bounding boxes, while other video grounding datasets use trimmed videos or focus on people.

8. Baseline Details

The baselines combine temporal clip localization with frame- or tube-level grounding, while retaining the pre-generated-tube limitations that motivate STGRN.

  • Six baselines combine GroundeR, STPR, or WSSTG with either TALL or L-Net for temporal and spatio-temporal localization.Three additional baselines use ground-truth temporal clips.
  • TALL samples candidate clips with a sliding window and ranks them using clip-sentence scores and context features.
  • L-Net models evolving frame-by-word interactions and dynamically aggregates matching evidence to localize temporal boundaries.
  • GroundeR grounds language frame by frame and connects selected regions into a tube, but lacks temporal context modeling.
  • STPR and WSSTG use tube-level grounding but inherit the difficulty of pre-generating high-quality tubes without textual clues.They also model single tubes and ignore relationships between objects.
  • All baselines use the same TALL or L-Net models and are trained with Adam to ensure fair comparison.GroundeR, STPR, and WSSTG are trained within ground-truth clips and used after predicted clip localization at inference.

9. More Ablation Study

The ablations examine explicit graph construction and query-object modeling, showing that directed relations and object-aware query features are central design choices.

  • Directed GCN: The directed GCN outperforms an undirected GCN and GAT in the directed-GCN ablation.The undirected GCN and GAT perform close to the model without explicit subgraph modeling.
  • Query Modeling: The query model identifies the first noun or interrogative word corresponding to the queried object and derives an object-aware query vector with context attention.This differs from modeling the sentence only as a whole query vector with an RNN.
Loading 2001.06891v3…