PaperlayerView exact revision on arXiv

Source-linked AI summary

Human-centric Spatio-Temporal Video Grounding With Visual Transformers

Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, Dong Xu

arXiv:2011.05049v2cs.CVcs.AIcs.MM

TL;DR

Existing grounding tasks do not jointly localize a described person in space and time within an untrimmed video, despite applications involving long surveillance footage. The paper introduces HC-STVG, an annotated benchmark, and STGVT, which uses Visual Transformers for video-text grounding; STGVT outperforms the reported baselines on HC-STVG.

  • Problem

    HC-STVG addresses the gap between spatial-only image grounding and temporal-only video grounding by localizing a target person’s spatio-temporal tube in an untrimmed video.

  • Method

    STGVT generates tube proposals, models tube-sentence interactions with a Visual Transformer, predicts the matching tube, and trims irrelevant frames.

  • Results

    STGVT achieves 18.15% m vIoU and outperforms baseline methods, including on vIoU@0.3 and vIoU@0.5.

  • Takeaways & Limitations

    HC-STVG provides a benchmark for human-centric spatial-temporal video grounding, while STGVT establishes an effective baseline for the task.

Abstract

from arXiv · show

In this work, we introduce a novel task - Humancentric Spatio-Temporal Video Grounding (HC-STVG). Unlike the existing referring expression tasks in images or videos, by focusing on humans, HC-STVG aims to localize a spatiotemporal tube of the target person from an untrimmed video based on a given textural description. This task is useful, especially for healthcare and security-related applications, where the surveillance videos can be extremely long but only a specific person during a specific period of time is concerned. HC-STVG is a video grounding task that requires both spatial (where) and temporal (when) localization. Unfortunately, the existing grounding methods cannot handle this task well. We tackle this task by proposing an effective baseline method named Spatio-Temporal Grounding with Visual Transformers (STGVT), which utilizes Visual Transformers to extract cross-modal representations for video-sentence matching and temporal localization. To facilitate this task, we also contribute an HC-STVG dataset consisting of 5,660 video-sentence pairs on complex multi-person scenes. Specifically, each video lasts for 20 seconds, pairing with a natural query sentence with an average of 17.25 words. Extensive experiments are conducted on this dataset, demonstrating the newly-proposed method outperforms the existing baseline methods.

I. INTRODUCTION

HC-STVG extends visual grounding to localize a described person’s spatio-temporal tube in an untrimmed video. The paper introduces this task, a benchmark dataset, and the STGVT baseline to address joint spatial-temporal grounding in complex scenes.

  • Task motivation: The task is challenging because the target may appear throughout a video while the described event occupies only a short interval.Determining temporal boundaries requires textual action and relation cues together with visual video cues.
  • Applications: HC-STVG targets healthcare and security scenarios requiring retrieval of a specific person’s behavior from extremely long surveillance videos.The desired output is a sequence of bounding boxes for the target person across frames matching the described action.
  • Task motivation: HC-STVG localizes the described person’s spatio-temporal tube, combining spatial and temporal grounding in an untrimmed video.Unlike image referring expression and temporal video grounding, it returns both the target person’s location and the relevant action interval.
  • Challenges: The task requires simultaneous spatial-temporal grounding, multimodal reasoning in multi-person scenes, and dynamic action-boundary detection.These challenges involve visual and textual attributes, actions, human-object relations, and fine-grained cross-modality reasoning.
  • Approach: STGVT uses a Visual Transformer to jointly exploit video and textual information for the new grounding task.The method is presented as an effective baseline for the proposed benchmark.
  • Dataset: The HC-STVG dataset contains 5,660 video-query pairs from complex multi-person scenes, with queries averaging 17.25 words.The dataset is intended as a benchmark for the human-centric task.
  • Contributions: The paper contributes a human-centric video-description dataset with spatio-temporal annotations and an effective baseline method evaluated through extensive experiments.These contributions establish the task, benchmark, and initial method for subsequent research.

II. RELATED WORK

Prior grounding work separately localizes objects in images or temporal segments in videos. HC-STVG instead addresses simultaneous spatial and temporal localization of a target person using a benchmark with complex human interactions.

  • Referring Expression: Referring Expression methods localize an object described by natural language, typically selecting among visual object proposals.Earlier approaches generated object proposals and matched them with textual descriptions; MAttNet decomposed expressions into subject, location, and relation.
  • Visual Transformer: Visual Transformer methods learn multimodal representations from image-text or video-text pairs and predict regions of interest.Existing approaches include single-stream and two-stream transformer models.
  • HC-STVG benchmark: The HC-STVG dataset provides video descriptions, bounding boxes, and temporal annotations for human targets in complex multi-person scenes.Its examples include both spatial annotations and temporal annotations, with semi-automatic tracking used for remaining frames.
  • Temporal Video Grounding: Temporal video grounding methods localize the video segment most relevant to a query sentence.Prior approaches generate temporal candidates or query-specific proposals and select the relevant segment using video-text interactions.
  • HC-STVG benchmark: Unlike prior video grounding work, HC-STVG performs spatial and temporal localization simultaneously for a target person.The task focuses on humans rather than arbitrary object categories.

A. Task Formulation

HC-STVG maps an untrimmed video and natural-language description to a target person’s spatio-temporal tube. Its benchmark uses normalized videos, rich interaction descriptions, and annotations constructed through manual keyframes plus tracking.

  • A. Task Formulation: Given an untrimmed video and description of actions involving a specific person, the system predicts the target segment and bounding boxes across that segment.The output is a sequence of bounding boxes forming the spatio-temporal tube.
  • B. Overview of HC-STVG dataset: The dataset contains 5,660 video-sentence pairs, including 4,500 training pairs and 1,160 testing pairs.Ground-truth tubes average 5.37 seconds, while videos are normalized to 20 seconds.
  • B. Overview of HC-STVG dataset: HC-STVG contains precise human-centric spatio-temporal annotations in complex multi-person scenes.57.2% of videos contain more than three people, and the remaining videos contain two people.
  • B. Overview of HC-STVG dataset: Its descriptions emphasize human-human and human-object interactions, with 56.1% containing both interaction types.The sentences also describe rich action sequences involving the target person.
  • C. Dataset Construction: Dataset construction comprises raw-video preparation, span selection, description, bounding-box annotation, and span extension.These five modules define the collection pipeline.
  • C. Dataset Construction: Raw videos are filtered from AVA-derived YouTube material selected for diverse, high-quality realistic scenes and interactions.The source material supports creating video-description pairs centered on character actions and interactions.
  • C. Dataset Construction: Annotators select describable multi-person spans without shot changes and with a minimum duration of 3 seconds.These rules increase grounding difficulty while preserving semantic coherence.
  • C. Dataset Construction: To reduce annotation effort, annotators label three keyframes and SiamRPN tracks the target person in the remaining frames.Video spans are then extended around the temporal ground truth to produce fixed-length query videos and negative temporal context.

D. Comparison with the Existing Datasets

HC-STVG combines spatial and temporal localization of a target person, while the proposed STGVT framework generates, matches, selects, and trims candidate tubes using video and language.

  • Dataset comparison: HC-STVG targets spatio-temporal localization of a person from a video-language pair, unlike datasets focused only on temporal or spatial localization.The section’s supplied passages describe both the task distinction and the STGVT framework stages.
  • Cross-modal representation: A visual transformer processes each tube-description pair to produce cross-modal features for downstream prediction.The supplied framework description places visual-transformer processing after proposal generation.
  • Tube proposals generation: The framework first generates tube proposals by linking bounding boxes in consecutive frames.This proposal-generation stage groups detections into candidate spatio-temporal tubes.
  • Tube selection and trimming: The framework predicts matching and classification scores plus tube offsets, which support tube selection and trimming.These outputs correspond to the framework’s later selection and trimming modules.

B. Cross-modal Representation

STGVT uses a visual transformer to model interactions between a query sentence and each proposed tube, producing global and frame-level cross-modal representations.

  • Cross-modal representation: The visual transformer separately receives textual and visual inputs and models their interaction through co-attention layers.Textual inputs include position, token, and segment information; visual inputs include temporal position, appearance, and spatial location.
  • Cross-modal representation: Textual inputs encode query position, tokens, and segments, while visual inputs encode frame indices, bounding-box features, and locations.Bounding-box coordinates are represented through location embeddings.
  • Cross-modal representation: The transformer outputs a global feature for each tube-sentence pair and frame features for each bounding box in the tube.These representations support tube-level and frame-level matching predictions.

C. Tube-description Matching

Tube-description matching scores candidate proposals with a binary classifier and relaxes positive-sample labeling using frame overlap and average IoU criteria.

  • Tube-description matching: The global cross-modal feature feeds a binary classifier that predicts a matching score for each tube-description pair.The proposal with the highest matching score is selected for the final processing stage.
  • Positive-sample labeling: Positive tube samples satisfy both soverlap >= 0.9 and sIoU > 0.5.This relaxed labeling strategy addresses imbalance between positive and negative samples.
  • Positive-sample labeling: soverlap measures the ratio of intersected frames between a proposal and ground-truth tube relative to all ground-truth frames.The supplied passage defines the score through the intersection of proposal and ground-truth frames.
  • Positive-sample labeling: A tube is treated as negative when its sIoU is lower than 0.2.The average IoU compares detected and ground-truth bounding boxes across corresponding frames.

D. Tube Trimming

STGVT trims selected proposals by scoring frame relevance and regressing temporal boundary offsets, with training losses supervising matching, classification, and regression.

  • Tube trimming: Tube trimming removes redundant transition frames from the selected proposal to produce the final predicted tube.The module computes frame-query relevance and predicts temporal offsets.
  • Classification subnet: A classification subnet predicts whether each proposal frame should remain in the final spatio-temporal localization result.It uses each frame’s cross-modal feature to produce a relevance score.
  • Boundary regression subnet: A boundary-regression subnet predicts each frame’s offsets from the ground-truth temporal start and end boundaries.The two outputs represent left and right temporal offsets.
  • Loss functions: The total loss combines tube matching, frame classification, and boundary regression objectives.The regression objective uses IoU loss, while classification and matching use cross-entropy losses; classification and regression losses apply to positive tubes and frames.

F. Inference

During inference, STGVT ranks tube–sentence pairs, selects the highest-scoring tube, and trims its temporal extent using frame-level predictions.

  • F. Inference: The visual transformer extracts cross-modal global and frame features for each tube proposal and query-sentence pair.These features support both tube matching and frame-level temporal localization.
  • F. Inference: The tube-description matching module predicts a matching score for each proposal, and the highest-scoring tube is selected for trimming.The matching score is based on the global feature.
  • F. Inference: For each frame in the selected tube, classification scores and regression offsets predict temporal boundaries.The method initializes a range from the highest-scoring frame and merges overlapping predicted ranges whose classification scores exceed a threshold.

V. EXPERIMENTS

The experiments evaluate predicted spatio-temporal tubes with vIoU-based metrics and report implementation settings for the HC-STVG evaluation.

  • V. EXPERIMENTS: vIoU measures predicted spatio-temporal tube accuracy using the intersection and union of predicted and ground-truth tubes.The reported metrics include vIoU@0.3, vIoU@0.5, and mean vIoU over the test set.
  • V. EXPERIMENTS: vIoU@perc. is the percentage of predicted tubes whose vIoU exceeds perc., while mvIoU averages vIoU across the test set.These metrics assess overlap at fixed thresholds and overall average overlap.
  • V. EXPERIMENTS: During inference, the method samples one frame every six frames, limits detections to 101 bounding boxes per image, and truncates or pads queries to 40 words.Visual features are extracted with Mask R-CNN pretrained on Visual Genome.

B. Baseline Methods

The experiments compare STGVT with composed temporal-and-spatial grounding baselines and analyze tube matching, trimming, and pretraining components.

  • B. Baseline Methods: TALL and 2D-TAN first generate temporal segments, after which WSSTG localizes spatio-temporal tubes within those segments.These combinations provide baselines for applying existing grounding methods to untrimmed videos.
  • C. Experimental Results: STGVT achieves an mvIoU score of 18.15% and surpasses baseline methods on vIoU@0.3 and vIoU@0.5.The comparison is reported on the HC-STVG dataset.
  • C. Experimental Results: STGVT also demonstrates effectiveness on the non-human-centric VidSTG dataset.The experiment considers all categories when generating tubes and applies the same tube-selection and trimming procedure.
  • D. Ablation Study: The full STGVT method significantly outperforms its version without tube trimming, while the untrimmed version still outperforms other baseline methods.The study also reports that the selected trimmed positive tube outperforms the baseline method.
  • E. Qualitative Analysis: In representative examples, STGVT improves temporal localization in the 2nd and 5th examples and spatial localization in the 1st, 3rd, and 5th examples.The qualitative comparison is against WSSTG+TALL.
  • D. Ablation Study: STGVT without pretraining still outperforms existing baselines, while pretraining the visual transformer further improves performance.The pretraining uses the Conceptual Captions dataset.

E. Qualitative Analysis

Qualitative results show that STGVT more closely matches temporal ground truth and often produces more accurate spatial boxes than the baseline.

  • E. Qualitative Analysis: In most examples, STGVT’s temporal tube predictions overlap the ground truth more than TALL+WSSTG predictions.The comparison indicates more precise localization of the relevant video segment.
  • E. Qualitative Analysis: In the third example, both methods achieve good temporal grounding, but STGVT generates more accurate bounding boxes.This separates spatial improvement from cases where temporal localization is already comparable.
  • VI. CONCLUSION: The paper concludes that STGVT outperforms existing grounding methods for the HC-STVG task.The conclusion follows experiments on the introduced human-centric spatio-temporal grounding setting.
Loading 2011.05049v2…