PaperlayerView exact revision on arXiv

Source-linked AI summary

Local-Global Video-Text Interactions for Temporal Grounding

Jonghwan Mun, Minsu Cho, Bohyung Han

arXiv:2004.07514v1cs.CV

TL;DR

Text-to-video temporal grounding seeks the video interval semantically relevant to a text query. The paper proposes a regression-based model that extracts multiple semantic-phrase features and performs local-global video-text interactions. It achieves state-of-the-art performance on Charades-STA and ActivityNet Captions.

  • Problem

    Text-to-video temporal grounding must localize the video interval corresponding to a text query, while existing methods can miss phrase-level details or comprehensive context.

  • Method

    The model uses sequential query attention to extract multiple semantic phrases, then aligns them with video segments through three-level local-global interactions before regressing the interval.

  • Results

    The method achieves state-of-the-art performance on both Charades-STA and ActivityNet Captions datasets.

  • Takeaways & Limitations

    Ablation results identify both local and global context in video-text interactions as crucial to accurate grounding.

  • Takeaways & Limitations

    The method can fail on confusing videos where similar visual events occur at multiple time intervals.

Abstract

from arXiv · show

This paper addresses the problem of text-to-video temporal grounding, which aims to identify the time interval in a video semantically relevant to a text query. We tackle this problem using a novel regression-based model that learns to extract a collection of mid-level features for semantic phrases in a text query, which corresponds to important semantic entities described in the query (e.g., actors, objects, and actions), and reflect bi-modal interactions between the linguistic features of the query and the visual features of the video in multiple levels. The proposed method effectively predicts the target time interval by exploiting contextual information from local to global during bi-modal interactions. Through in-depth ablation studies, we find out that incorporating both local and global context in video and text interactions is crucial to the accurate grounding. Our experiment shows that the proposed method outperforms the state of the arts on Charades-STA and ActivityNet Captions datasets by large margins, 7.44\% and 4.61\% points at Recall@tIoU=0.5 metric, respectively. Code is available in https://github.com/JonghwanMun/LGI4temporalgrounding.

1. Introduction

Text-to-video temporal grounding localizes the video interval corresponding to a text query. The paper extracts multiple semantic phrases and models local-global video-text interactions to improve regression-based localization.

  • Text-to-video temporal grounding aims to localize the video time interval corresponding to a text query.
  • Semantic phrases represent entities such as actors, objects, actions, or places and can correspond to multiple parts of a target interval.The example contains an actor phrase and three action phrases.
  • Existing scan-and-localize methods compare candidate proposals with a single global query feature, which can miss phrase-level localization details.An attention-based regression method identifies a discriminative phrase but does not capture comprehensive context.
  • The proposed method extracts multiple distinct semantic phrase representations with sequential query attention for subsequent video-text interaction.
  • Local-global video-text interactions model relationships between video segments and semantic phrases at multiple levels before regression-based localization.
  • The method outperforms state-of-the-art methods by a large margin on both Charades-STA and ActivityNet Captions.

2. Related Work

Prior temporal grounding methods mainly scan candidate clips or directly regress boundaries. The paper positions its approach against these alternatives by combining phrase-aware modeling with multi-level video-text interactions.

  • Temporal action detection methods use dense frame prediction, proposal refinement, or single-shot detection to determine action intervals.
  • Text-to-video grounding differs from single-action detection because a text query may require localizing complex intervals involving multiple actions.
  • Scan-and-localize grounding scans videos with sliding windows and selects the candidate clip with the highest query-matching score.The approach can be time-consuming and generate redundant candidates.
  • Proposal-free methods directly identify video-segment start and end indices, including query-guided filtering and attention-based location regression.
  • Compared with ABLR, the proposed method models multiple semantic phrases and uses multi-level interactions to capture finer video-text correlations.
  • The overall architecture encodes segment-level visual and sentence-level textual features before extracting phrase features and performing local-global interactions.

3. Proposed Method

The proposed method represents queries as multiple semantic phrases and aligns them with video segments through local-global video-text interactions. It combines segment-level fusion, local context, global phrase-relation modeling, and temporal attention to regress the target interval.

  • Encoders: The method encodes video segments with visual features and timestamps, while a bidirectional LSTM produces word- and sentence-level query representations.Temporal position encoding supports identifying semantics at diverse temporal locations during location regression.
  • Sequential Query Attention Network (SQAN): SQAN sequentially attends over word-level features to extract multiple semantic phrase representations without phrase-level ground-truth annotations.Each step conditions on preceding phrase features to attend to different query aspects and represent entities such as actors, objects, and actions.
  • Local-Global Video-Text Interactions: Each semantic phrase interacts with segment features through Hadamard-product fusion and local temporal convolutions that capture neighboring context.The local residual block uses large-bandwidth kernels to cover long-range semantic entities beyond an individual 16-frame segment.
  • Local-Global Video-Text Interactions: The model aggregates phrase-specific segment features with attentive pooling and applies a non-local block to model global temporal relations between semantic phrases.Global context helps connect phrases with large temporal gaps, such as resolving a pronoun using an earlier phrase.
  • Prediction: Temporally attended semantics-aware segment features are passed to an MLP that predicts the start and end times of the target interval.The overall architecture combines sequential query attention, multi-level interactions, and temporal attention before regression.

4.1. Datasets

The evaluation uses Charades-STA and ActivityNet Captions, which differ substantially in size, video duration, and annotation structure.

  • Charades-STA: Charades-STA contains 12,408 training and 3,720 test time-interval/query pairs.Videos average 30 seconds, and queries contain at most 10 words.
  • ActivityNet Captions: ActivityNet Captions contains 20k YouTube videos averaging 120 seconds, split into training, validation, and testing sets.The split includes 10,024 training, 4,926 validation, and 5,044 testing videos.
  • ActivityNet Captions: ActivityNet Captions videos contain 3.65 localized intervals and descriptions averaging 13.48 words.Performance is reported on the combined two validation sets, denoted val 1 and val 2.

4.2. Metrics

Performance is evaluated with temporal overlap and localization-quality metrics, while the ActivityNet Captions comparison table highlights the best results.

  • Metrics: Recall at various temporal Intersection over Union thresholds measures the percentage of predictions exceeding each threshold.The reported thresholds are 0.3, 0.5, and 0.7.
  • Metrics: Mean averaged tIoU (mIoU) is used as a second performance-comparison metric.It complements thresholded Recall by summarizing temporal overlap quality.
  • Metrics: Table 2 compares algorithms on ActivityNet Captions, with bold-faced numbers indicating the best performance.The table presents comparative results rather than a dataset description.

4.3. Implementation Details

Implementation uses fixed pretrained 3D CNN features, uniform video segmentation, dataset-specific vocabularies, and query-length truncation for ActivityNet Captions.

  • Video features: I3D extracts segment features for Charades-STA, while C3D extracts them for ActivityNet Captions.The CNN parameters remain fixed during training.
  • Video features: Each video is uniformly sampled into T = 128 segments.This provides the segment sequence used by the model.
  • Query encoding: Query encoding retains lower-cased, tokenized words and uses vocabularies of 1,140 and 11,125 words for the two datasets.ActivityNet Captions queries are truncated to a maximum of 25 words.

4.4. Comparison with Other Methods

The comparison covers scan-and-localize and proposal-free methods, and the proposed algorithm surpasses competing methods on both benchmark datasets.

  • Compared methods: Compared methods include scan-and-localize algorithms such as MCN, CTRL, SAP, ACL, ACRN, MLVI, TGN, MAN, TripNet, SMRL, and RWM.The comparison also includes proposal-free methods.
  • Compared methods: Proposal-free baselines include ABLR, ExCL, and PfTML-GA.These methods form the second comparison group.
  • Results: 7.44% and 4.61% points are the proposed method’s gains over state-of-the-art performance in R@0.5 on Charades-STA and ActivityNet Captions, respectively.Tables 1 and 2 report that the algorithm outperforms all competing methods.

4.5. In-Depth Analysis

The ablations show that semantic-phrase extraction, appropriate SQAN hyperparameters, and explicit local-global context modeling each improve temporal grounding. The strongest gains arise when local and global context are combined, while qualitative comparisons favor the full LGI model.

  • Main Ablation Studies: Semantic-phrase extraction and its distinctiveness and temporal-guidance losses improve localization over sentence-level query representations.LGI outperforms LGI–SQAN, while Ldqa and Ltag each provide additional benefits.
  • Main Ablation Studies: Performance increases up to 3 semantic phrases on Charades-STA and 5 on ActivityNet Captions, then decreases as phrases become too short.The authors attribute the decline to fragmented phrases that fail to describe proper semantics.
  • Main Ablation Studies: λ values of 0.2 and 0.3 generally perform well, whereas higher values make the model focus on one or two words as phrases.λ controls the distinct query attention loss Ldqa.
  • Analysis on Local-Global Video-Text Interaction: Combining local and global context modeling produces the best performance gain of 16.48% points.Local or global modeling alone improves alignment, but explicit local modeling combined with global modeling is strongest.
  • Analysis on Local-Global Video-Text Interaction: Hadamard-product fusion performs best, and early segment-level fusion leads to better accuracy than later fusion options.The authors associate Hadamard product with a gating operation and position embeddings with identifying entities at diverse temporal locations.
  • Qualitative Results: The full LGI model predicts more accurate locations than LGI–SQAN through query understanding at the semantic-phrase level.The comparison is illustrated with predictions and temporal attention weights.

5. Conclusion

The paper presents a local-global video-text interaction algorithm for text-to-video temporal grounding through constituent semantic-phrase extraction. It reports state-of-the-art performance on both Charades-STA and ActivityNet Captions.

  • 5. Conclusion: The proposed multilevel interaction scheme captures relationships between semantic phrases and video segments by modeling local and global contexts.The approach performs temporal grounding via constituent semantic-phrase extraction.
  • 5. Conclusion: The algorithm achieves state-of-the-art performance on both Charades-STA and ActivityNet Captions datasets.

6. Supplementary Material

The supplementary material describes the sentence-level LGI–SQAN variant, adds qualitative visualizations, and documents failure cases. It contrasts whole-query interaction with explicit phrase extraction and shows both attention behavior and localization errors.

  • Supplementary Architecture: LGI–SQAN performs local-global interactions using a sentence-level feature representing the whole query rather than explicitly extracted semantic phrases.
  • Supplementary Architecture: The LGI–SQAN pipeline copies the sentence feature across temporal segments, applies segment fusion and local-global modeling, then regresses [ts, te].Local context uses either a masked non-local block or a residual block.
  • Qualitative Results: The qualitative visualizations report temporal attention, query attention, and predictions for Charades-STA and ActivityNet Captions.T-ATT denotes temporal attention weights, while Q-ATT denotes query attention weights.
  • Failure Cases: One failure case involves confusing videos where a man appears to smile at multiple time intervals, although attention still captures relevant segments at diverse locations.
  • Failure Cases: Another failure occurs when the model separates “wooden” and “floorboards” instead of extracting the natural phrase “wooden floorboards,” causing inaccurate localization.
  • Failure Cases: Figure 10 presents failure examples from the Charades-STA and Activity Captions datasets.
Loading 2004.07514v1…