PaperlayerView exact revision on arXiv

Source-linked AI summary

UniVTG: Towards Unified Video-Language Temporal Grounding

Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, Mike Zheng Shou

arXiv:2307.16715v2cs.CV

TL;DR

VTG methods often use task-specific labels and models, limiting coverage across varied grounding tasks while fine-grained temporal annotations remain costly. UniVTG unifies point-, interval-, and curve-level labels, creates scalable pseudo supervision, and pretrains a flexible grounding model; across three tasks and seven datasets, it reports strong results including zero-shot grounding.

  • Problem

    Task-specific VTG models and labels limit generalization across moment retrieval, highlight detection, and video summarization, while fine-grained temporal annotation is expensive to scale.

  • Method

    UniVTG unifies point-, interval-, and curve-level labels, uses CLIP to create scalable pseudo temporal labels, and pretrains one flexible grounding model across tasks.

  • Results

    Across three VTG tasks and seven datasets, UniVTG outperforms methods tailored to individual tasks and supports zero-shot grounding after large-scale pretraining.

  • Takeaways & Limitations

    Grounding can serve as a shared pretraining task for diverse temporal labels and transfer to multiple VTG tasks, including zero-shot settings.

Abstract

from arXiv · show

Video Temporal Grounding (VTG), which aims to ground target clips from videos (such as consecutive intervals or disjoint shots) according to custom language queries (e.g., sentences or words), is key for video browsing on social media. Most methods in this direction develop taskspecific models that are trained with type-specific labels, such as moment retrieval (time interval) and highlight detection (worthiness curve), which limits their abilities to generalize to various VTG tasks and labels. In this paper, we propose to Unify the diverse VTG labels and tasks, dubbed UniVTG, along three directions: Firstly, we revisit a wide range of VTG labels and tasks and define a unified formulation. Based on this, we develop data annotation schemes to create scalable pseudo supervision. Secondly, we develop an effective and flexible grounding model capable of addressing each task and making full use of each label. Lastly, thanks to the unified framework, we are able to unlock temporal grounding pretraining from large-scale diverse labels and develop stronger grounding abilities e.g., zero-shot grounding. Extensive experiments on three tasks (moment retrieval, highlight detection and video summarization) across seven datasets (QVHighlights, Charades-STA, TACoS, Ego4D, YouTube Highlights, TVSum, and QFVS) demonstrate the effectiveness and flexibility of our proposed framework. The codes are available at https://github.com/showlab/UniVTG.

1. Introduction

UniVTG frames moment retrieval, highlight detection, and video summarization as one Video Temporal Grounding problem: grounding varied-scale clips from customized queries. It unifies their labels, creates scalable pseudo supervision, pretrains one grounding model, and transfers it across tasks and datasets.

  • Motivation: Video Temporal Grounding unifies moment retrieval, highlight detection, and video summarization as grounding varied-scale clips from customized language queries.The tasks target consecutive intervals, key segments, or disjoint shots using sentences, titles, keywords, or other queries.
  • Motivation: Task-specific labels and models limit generalization across VTG tasks, while fine-grained temporal annotation is expensive and difficult to scale.Prior unified efforts focus on only two subtasks and rely on specific labels.
  • Approach: UniVTG defines a unified formulation that assigns three query-conditional elements to each clip, covering point-, interval-, and curve-level supervision.The formulation is designed to unify diverse VTG labels and tasks within one framework.
  • Approach: A CLIP-based annotation scheme produces scalable fine-grained pseudo labels, enabling temporal grounding pretraining from large-scale diverse annotations.The framework uses labels from varied sources and converts available supervision into a common representation.
  • Approach: UniVTG uses one flexible grounding model with modality-fusion and modality-alignment pathways plus three heads, supporting each task and label type.The unified model is intended to make full use of the available supervision rather than requiring task-specific models.
  • Evaluation: Experiments cover three VTG tasks across seven datasets, and the 4.2M-sample pretrained model outperforms methods tailored to individual tasks.The framework also demonstrates zero-shot grounding and reports detailed ablations of its components.

3. Towards Unified VTG: Tasks and Labels

UniVTG represents each video as query-conditioned clips with foreground, boundary, and saliency elements, then maps diverse grounding tasks into this shared formulation. It also uses scalable supervision schemes to derive interval, curve, and point labels for moment retrieval, highlight detection, and video summarization.

  • Unified Formulation: Each video is divided into fixed-length clips, each assigned a foreground indicator, boundary offsets, and query-conditioned saliency score.The formulation represents interval boundaries through distances from each clip's centered timestamp.
  • Moment Retrieval and Interval-wise Label: Moment retrieval selects one or more consecutive clip sets corresponding to sentence queries and can be represented by foreground clip boundaries.Interval labels are converted by setting foreground and saliency values for clips inside and outside target intervals.
  • Highlight Detection and Curve-wise Label: Highlight detection ranks clips by importance, using video titles or domain names as queries when language queries are unavailable.CLIP-derived similarities provide pseudo curve labels by selecting the top-5 concepts as the video gist.
  • Highlight Detection and Curve-wise Label: CLIP temporal-label generation uses a concept bank, clip-level concept scores, and thresholding to derive saliency, interval, and point labels.Clips above threshold τ receive foreground indicators, while offsets use distances to neighboring non-foreground clips.
  • Video Summarization and Point-wise Label: Query-focused video summarization selects query-relevant clips as a concise set whose size does not exceed α% of the original video.Point labels indicate whether each shot belongs to a queried concept, and narration timestamps can provide scalable supervision.

4. Towards Unified VTG: Model

UniVTG uses a unified grounding model that encodes video and text, aligns and interacts across modalities, and decodes three query-conditional elements through specialized heads. The same outputs are converted into predictions for moment retrieval, highlight detection, and video summarization.

  • The model combines frozen video and text encoders with a multi-modal encoder containing k self-attention blocks and three prediction heads.Video features concatenate CLIP and SlowFast representations, while the text encoder extracts token-level CLIP features.
  • Video and text features are projected into a shared dimension, then processed through separate cross-modal alignment and interaction pathways.Alignment uses attentive query pooling for contrastive learning, while interaction adds positional and modality-type embeddings before transformer processing.
  • Three heads predict foreground matching, temporal boundaries, and clip saliency to correspond with the unified elements.The foreground head uses convolution and binary cross-entropy; the boundary head predicts left and right offsets with smooth L1 and generalized IoU losses; saliency uses video-text cosine similarity.
  • Training combines the losses from all three heads over clips, with saliency supervision using both intra-video and inter-video contrastive learning.Intra-video negatives are less-salient clips from the same video, while inter-video negatives are sentences from other batch samples.
  • At inference, moment retrieval applies one-dimensional NMS to predicted boundaries, highlight detection ranks foreground-plus-saliency scores, and summarization returns the top 2% foreground clips.Moment retrieval uses an NMS threshold of 0.7; summarization first segments videos with KTS.

5. Experiments

Experiments evaluate UniVTG across joint and individual temporal-grounding tasks, showing gains from unified pretraining and diverse label corpora. The model also supports zero-shot grounding across several benchmarks.

  • Experimental scope: UniVTG is evaluated across four VTG settings and seven datasets, covering joint retrieval/highlight detection, moment retrieval, highlight detection, and summarization.The experiments address the effects of grounding pretraining, label diversity, and the unified formulation and model.
  • Joint moment retrieval and highlight detection: +8.16 Avg. mAP and +5.32 HIT@1 are gained on QVHighlights with large-scale pretraining, surpassing all baselines by a large margin.Without pretraining, UniVTG remains comparable to joint optimization methods; with pretraining, it improves on all metrics.
  • Moment retrieval: +2.97 mIoU in NLQ, +2.07 in Charades-STA, and +5.03 in TACoS result from large-scale grounding pretraining.Zero-shot performance surpasses all baselines in NLQ but is inferior on TACoS, where visually similar scenes make zero-shot grounding difficult.
  • Highlight detection: Grounding pretraining improves UniVTG and produces higher Avg. mAP than all baselines for highlight detection on YouTube Highlights and TVSum.UniVTG also beats several video-only baselines in the zero-shot setting; gains are more consistent on the larger YouTube dataset than on TVSum.
  • Video summarization: 0.8% higher Avg. F1-score than IntentVizor is achieved on QFVS by pretrained UniVTG.IntentVizor is an interactive method tailored to video summarization, while this result evaluates UniVTG’s generalization to that task.
  • Pretraining ablations: Combining point-, interval-, and curve-label corpora further boosts pretraining outcomes, including +5.2 MR’ performance.The corpus contains 1.8M point labels, 0.9M interval labels, and 1.5M pseudo curve labels, totaling 4.2M temporal annotations.

A. CLIP teacher strategy

The CLIP teacher strategy derives video concepts and temporal curves, then converts those curves into flexible intervals using an adaptive threshold.

  • Concept bank: The concept bank contains 19,995 common class names used to identify salient video concepts.The list includes examples such as “Sandwich Cookies,” “Air conditioning,” and “Advertising.”
  • CLIP teacher: Frame-level clip image features sampled at 2 fps are compared with the class list, and the top-5 classes by average similarity represent video concepts.These similarity scores provide the basis for deriving temporal curve labels.
  • Curve-to-interval conversion: The maximum discrete curve value after 0.05 discretization becomes the threshold, and adjacent clips sharing it form one or more temporal windows.This avoids a fixed threshold and permits multiple intervals from a single video.

B. Datasets

UniVTG builds its pretraining corpus from timestamped narrations, captioned intervals, and CLIP-derived curves, then evaluates across diverse VTG benchmarks.

  • Pretraining corpus: Point labels come from timestamped Ego4D narrations after excluding NLQ validation and test splits.These labels associate narrations with exact video timestamps.
  • Pretraining corpus: Interval labels use start and end timestamps with captions from fewer than 300K selected VideoCC videos.The captions serve as language queries for the temporal windows.
  • Pretraining corpus: Curve labels are derived from the selected VideoCC subset using the CLIP teacher method.The resulting corpus combines point-, interval-, and curve-level temporal supervision.
  • Benchmarks: QVHighlights benchmarks joint moment retrieval and highlight detection using 10,148 videos and 10,310 queries associated with 18,367 moments.The videos average 150 seconds and cover daily vlogs, travel vlogs, and news events.
  • Benchmarks: Charades-STA, Ego4D NLQ, and TACoS evaluate moment retrieval across indoor, egocentric, and other video scenarios.NLQ uses long daily-egocentric videos with question-like queries.

C. Experimental settings

Experimental settings tune task-specific parameters and preprocessing choices for highlight detection and summarization while maintaining class-balance and evaluation procedures.

  • Parameter selection: The highlight-detection parameter λintra is tuned separately within each YouTube Highlights and TVSum domain before aggregating results.The optimal domain-specific settings are listed in Tables 9 and 10.
  • Training: Negative samples with fi = 0 receive weight 0.1 in the binary cross-entropy loss to balance positive and negative samples.
  • Highlight inference: QVHighlights typically performs better with fi = 1 during highlight inference, whereas smaller datasets use f̃i for more reliable predictions.The smaller datasets are YouTube Highlights and TVSum.
  • Video summarization: QFVS evaluation aggregates foreground scores by segment and matches summaries using conceptual similarity based on intersection-over-union of clip concepts.Precision, recall, and F1 scores are then computed from the maximum-weight bipartite matching.

D. Ablation studies of training objective

The ablations separate downstream and pretraining objectives to assess their effects under the unified training setup.

  • D. Ablation studies of training objective: The study compares downstream objectives without pretraining and pretraining objectives while keeping downstream tuning at the same optimal settings.These comparisons are reported on the QVHighlights, NLQ, and TACoS validation splits.

E. Parameters sensitivity

Parameter sensitivity is examined through ablations of multi-modal encoder depth and projector dimension, with results visualized for retrieval and highlight detection.

  • E. Parameters sensitivity: Multi-modal encoder depth is ablated over L ∈ [1, 2, 3, 4, 6, 8] without pretraining.The experiment uses the QVHighlights validation split.
  • E. Parameters sensitivity: Projector dimension is varied from 256 to 1024 without pretraining.The study is presented in Figure 10 on the QVHighlights validation split.
  • E. Parameters sensitivity: The sensitivity results report Avg. mAP for moment retrieval and HIT@1 for highlight detection.These metrics correspond to the two plotted evaluation outcomes.

F. Loss weights

The effect of foreground loss weighting is evaluated on three moment retrieval benchmarks with pretraining.

  • F. Loss weights: Foreground loss weight λ_f is studied on QVHighlights, NLQ, and TACoS moment retrieval benchmarks.The corresponding ablation is reported in Table 13.

G. Visualizations

The paper provides quantitative visualizations of UniVTG predictions across different settings and domains.

  • G. Visualizations: Figures 7 and 8 visualize UniVTG predictions quantitatively across different settings and domains.The visualizations focus on prediction behavior rather than a separately stated numerical comparison.
Loading 2307.16715v2…