PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffusionVMR: Diffusion Model for Joint Video Moment Retrieval and Highlight Detection

Henghao Zhao, Kevin Qinghong Lin, Rui Yan, Zechao Li

arXiv:2308.15109v2cs.CV

TL;DR

Ambiguous temporal boundaries make it difficult to learn text-video correspondences for moment retrieval and highlight detection. DiffusionVMR treats both tasks as conditional denoising generation, and the paper reports a 12% average mAP improvement on QVHighlights over baseline.

  • Problem

    Ambiguous event boundaries in continuous video make text-video clip correspondences difficult to learn for moment retrieval and highlight detection.

    Source: §I · p. 1

  • Method

    DiffusionVMR jointly models moment retrieval and highlight detection as conditional denoising generation, training on corrupted ground truth and refining noise-initialized outputs at inference.

    Source: §I · p. 2

  • Results

    12% improvement in average mAP on QVHighlights over baseline is reported, with experiments spanning five datasets and both tasks.

    Source: §I · p. 2

  • Takeaways & Limitations

    Inference can use arbitrary settings without matching training, including different numbers of proposal spans for moment retrieval.

    Source: §I · p. 2, C. Ablation Studies · p. 9

Abstract

from arXiv · show

Video moment retrieval and highlight detection have received attention in the current era of video content proliferation, aiming to localize moments and estimate clip relevances based on user-specific queries. Given that the video content is continuous in time, there is often a lack of clear boundaries between temporal events in a video. This boundary ambiguity makes it challenging for the model to learn text-video clip correspondences, resulting in the subpar performance of existing methods in predicting target segments. To alleviate this problem, we propose to solve the two tasks jointly from the perspective of denoising generation. Moreover, the target boundary can be localized clearly by iterative refinement from coarse to fine. Specifically, a novel framework, DiffusionVMR, is proposed to redefine the two tasks as a unified conditional denoising generation process by combining the diffusion model. During training, Gaussian noise is added to corrupt the ground truth, with noisy candidates produced as input. The model is trained to reverse this noise addition process. In the inference phase, DiffusionVMR initiates directly from Gaussian noise and progressively refines the proposals from the noise to the meaningful output. Notably, the proposed DiffusionVMR inherits the advantages of diffusion models that allow for iteratively refined results during inference, enhancing the boundary transition from coarse to fine. Furthermore, the training and inference of DiffusionVMR are decoupled. An arbitrary setting can be used in DiffusionVMR during inference without consistency with the training phase. Extensive experiments conducted on five widely-used benchmarks (i.e., QVHighlight, Charades-STA, TACoS, YouTubeHighlights and TVSum) across two tasks (moment retrieval and/or highlight detection) demonstrate the effectiveness and flexibility of the proposed DiffusionVMR.

I. INTRODUCTION

DiffusionVMR reframes moment retrieval and highlight detection as joint conditional denoising-generation tasks, progressively refining noisy proposals and saliency scores. The paper reports improved benchmark performance and flexible inference settings.

I. INTRODUCTION

  • Moment retrieval localizes query-relevant temporal windows, whereas highlight detection selects salient video segments for a concise overview.The introduction describes proposal-based and proposal-free retrieval paradigms and query-tailored highlight detection.Source: §I · p. 1
  • DiffusionVMR addresses ambiguous temporal boundaries by jointly tackling retrieval and highlight detection through iterative coarse-to-fine denoising.The paper links boundary ambiguity to difficulty learning text-video clip correspondences.Source: §I · p. 2
  • Training corrupts ground truth with Gaussian noise for denoising, while inference starts from Gaussian noise and can use settings independent of training.The paper describes this decoupling as providing flexibility.Source: §I · p. 2
  • 12% improvement in average mAP on QVHighlights over baseline is reported, alongside experiments across five datasets.The evaluated datasets cover joint retrieval and highlight detection, individual moment retrieval, and individual highlight detection.Source: §I · p. 2
  • Moment retrieval starts spans from Gaussian noise and progressively refines them, while highlight detection similarly refines randomly initialized clip saliency scores.The retrieval branch uses noisy span candidates and the highlight branch generates a saliency vector during denoising.Source: §I · p. 2

II. RELATED WORK

Related work covers proposal-based and proposal-free moment retrieval, highlight scoring, and diffusion models in other tasks. DiffusionVMR applies conditional denoising generation to both query-grounded tasks.

A. Video Moment Retrieval

  • Proposal-based retrieval ranks pre-generated candidates, while proposal-free methods predict boundaries directly; DiffusionVMR instead initializes candidates from Gaussian noise and denoises them.The paper characterizes proposal-based methods as requiring storage for proposals and being sensitive to their manually designed scale.Source: A. Video Moment Retrieval · pp. 2–3

B. Highlight Detection

  • Traditional highlight datasets and methods are often query-agnostic, while QVHighlights supports query-tailored highlights and recent methods generally rank clips by saliency.DiffusionVMR contrasts with these ranking-based methods by using a generative diffusion perspective.Source: B. Highlight Detection · p. 3

C. Diffusion Models

  • Prior diffusion applications include detection, anomaly detection, pose estimation, and temporal action tasks, but moment retrieval and highlight detection had received little attention.The paper says its multimodal tasks are conditioned on distinct text queries, unlike the cited prior methods, and reports a performance advantage over contemporaneous MomentDiff.Source: C. Diffusion Models · p. 3

III. APPROACH

The approach casts temporal grounding as conditional denoising, using noisy samples that are progressively mapped toward target outputs. Its preliminary diffusion process explains how noise is added and reversed.

III. APPROACH

  • The framework treats moment retrieval and highlight detection as conditional denoising-generation tasks, iteratively refining predictions to address boundary ambiguity.The objective is to uncover relationships between text queries, video content, and event boundaries through iterative refinements.Source: §III · p. 3
  • For a video-text pair, the model predicts timestamp pairs for relevant moments and a saliency score for each video clip.A moment can be represented as a temporal span using its center and width.Source: §III · p. 3
  • In training, ground-truth moments undergo t diffusion steps before denoising; at inference, spans start from Gaussian noise and decoder layers iteratively update them.The described retrieval pipeline uses a cross-modal encoder and a moment denoising decoder.Source: Figure 2 · p. 4

A. Preliminary

  • The forward diffusion process adds Gaussian noise over steps until the sample distribution approaches Gaussian noise, and the reverse process maps noisy samples back toward the original.The noise rate is governed by α_t and a predefined variance schedule β_t; the reverse model estimates the original sample for iterative denoising.Source: A. Preliminary · p. 4

B. Model Overview

DiffusionVMR uses noisy moment spans and saliency vectors as inputs to separate denoising branches conditioned on video and text. At inference, both outputs are reconstructed from noise through iterative refinement.

B. Model Overview

  • The moment and saliency decoders denoise their respective noisy inputs using video-text representations, with moment predictions including spans and confidence scores.The moment branch uses query-based video representations, while the saliency branch uses video and text representations.Source: B. Model Overview · pp. 4–5
  • A randomly selected diffusion step adds Gaussian noise to padded, normalized ground-truth spans and progressively alters the ground-truth saliency vector.For datasets with explicit saliency labels, the average score per clip is used instead of one-hot labels.Source: B. Model Overview · p. 5
  • Inference starts from pure-noise proposal spans and saliency vectors, then denoises them through successive steps and uses the estimated x̂0 as the final result.Sampling steps may be skipped, and the inference proposal count need not match the training count.Source: B. Model Overview · p. 5

C. Moment Retrieval Branch

The moment retrieval branch builds query-conditioned video memory and progressively denoises proposal spans. Its decoder refines span features with temporal RoI alignment rather than learned queries.

C. Moment Retrieval Branch

  • The cross-modal encoder processes concatenated video and text features with six Transformer layers to produce contextual, query-based video representations for prediction.The resulting video memory encodes contextual semantics and cross-modal similarity relations.Source: C. Moment Retrieval Branch · p. 5
  • Two cascading denoising layers restore noisy spans by cropping corresponding video-memory features with temporal RoI alignment, interacting features, and updating the spans.Timestep embeddings indicate noise intensity to the denoising head, which uses a three-layer MLP; predicted spans encode temporal centre and width.Source: C. Moment Retrieval Branch · pp. 5–6
  • Unlike DETR, the decoder does not use learnable queries; it repeatedly refines proposal-span features with shared parameters, using each newly generated span at the next sampling stage.Iterative sampling regresses progressively more precise temporal boundaries.Source: C. Moment Retrieval Branch · p. 6

D. Highlight Detection Branch

The highlight detection branch generates one query-correlated saliency score vector across video clips. It combines generative and discriminative saliency signals.

D. Highlight Detection Branch

  • Unlike moment retrieval, which optimizes multiple noisy spans, highlight detection produces one saliency score vector covering all video clips.The vector reflects correlation between the query and each clip.Source: D. Highlight Detection Branch · p. 6
  • Attentive pooling aggregates query word embeddings into a sentence representation, which an enhanced cross-attention mechanism uses to reweight clip representations.The denoising decoder uses an MLP and a linear layer to compute the output distribution.Source: D. Highlight Detection Branch · p. 6
  • The branch retains the saliency calculation strategy from baseline and combines generative and discriminative scores as s = sgen + sdis.The discriminative strategy promotes interaction between textual words and video clips and is optimized with a discriminative loss.Source: D. Highlight Detection Branch · p. 6

E. Objective Function

The objective matches moment predictions to ground truths before computing a combined classification, span, and saliency loss. The saliency objective includes hinge and KL divergence terms.

E. Objective Function

  • The Hungarian algorithm finds an optimal bipartite assignment between moment predictions and ground truths before the loss is calculated.The assignment minimizes the matching cost between predictions and ground truths.Source: E. Objective Function · p. 6
  • The total loss combines classification, span, and saliency losses, with λclass and λsaliency weighting the classification and saliency terms.Classification measures foreground/background correctness, while span loss measures discrepancy between predicted and ground-truth moments.Source: E. Objective Function · p. 6
  • Span loss uses L1 and 1-D IoU terms, while saliency loss consists of hinge and KL divergence terms.The hinge term optimizes discriminative saliency scores, and KL divergence optimizes generative saliency scores.Source: E. Objective Function · pp. 6–7

IV. EXPERIMENTS

The experiments evaluate DiffusionVMR on both moment retrieval and highlight detection. The section covers benchmark comparisons, QVHighlights ablations, and qualitative results.

IV. EXPERIMENTS

  • The experiments assess DiffusionVMR on both video moment retrieval and highlight detection.The section proceeds through datasets, implementation details and metrics, state-of-the-art comparisons, QVHighlights ablations, and qualitative results.Source: §IV · p. 7
  • The evaluation includes comparisons with state-of-the-art methods, QVHighlights ablation studies, and qualitative results.Source: §IV · p. 7

A. Experiment Setting

Experiments use five benchmarks spanning joint moment retrieval and highlight detection, with task-specific datasets, features, and metrics. The implementation uses a shared model configuration and selects the best validation checkpoint.

A. Experiment Setting

  • QVHighlights supports joint retrieval and highlight evaluation, with 7,218 training, 1,550 validation, and 1,512 test query-moment pairs.Its test split is evaluated on a server.Source: A. Experiment Setting · p. 7
  • Charades-STA and TACoS provide moment-retrieval benchmarks, with 12,404 and 10,146 query-moment pairs used for training, respectively.Charades-STA has 3,720 test pairs; TACoS has 4,589 validation and 4,083 test pairs.Source: A. Experiment Setting · p. 7
  • TVSum and YouTube Highlights evaluate highlight detection; video titles serve as TVSum queries, while YouTube Highlights uses domain names.TVSum has 50 videos across 10 domains; YouTube Highlights has 433 videos across 6 domains.Source: A. Experiment Setting · p. 7
  • The model uses up to 75 frames and 32 query words, six cross-modal encoder layers, two denoising layers, and hidden dimension D = 256.Training lasts 300 epochs; inference proposal counts and sampling steps are selected by grid search.Source: A. Experiment Setting · p. 7
  • For video-moment retrieval, evaluation metrics include R1@M, mAP@M, and mAP@Avg; highlight detection uses mAP and Hit@1 on QVHighlights, mAP on TVSum, and Top-5 mAP on YouTube Highlights,.All metrics are higher-is-better; QVHighlights thresholds are Very Good.Source: A. Experiment Setting · p. 7

B. Comparison with State-of-the-arts

DiffusionVMR outperforms strong competitors on the reported joint and separate-task benchmarks, with gains in retrieval and highlight metrics. Its results are consistent across retrieval features and YouTube Highlights domains, while TVSum performance varies across domains.

Joint Moment Retrieval and Highlight Detection

  • DiffusionVMR improves average mAP by 2.88 and mAP@0.75 by +4 versus QD-DETR, and highlight mAP by +0.86 and HIT@1 by +3.03.On QVHighlights, it achieves the best performance across all reported metrics; the table uses 50 proposal spans and 5 sampling steps.Source: B. Comparison with State-of-the-arts · p. 7, Table I · p. 8

Moment Retrieval

Highlight Detection

C. Ablation Studies

The ablations examine inference proposal counts and refinement, model and loss components, training proposal matching, and robustness across random seeds. They report gains from increasing inference proposals and refinement, alongside component-specific contributions and marginal seed variation.

Number of Proposal Spans

  • DiffusionVMR improves steadily as inference proposals increase, while Moment-DETR degrades beyond 10 moment queries.The comparison uses mAP@Avg; more random proposals diversify retrieval initialization, and inference can vary proposal counts without retraining.Source: C. Ablation Studies · p. 9, Figure 4 · p. 9

Progressive Refinement

  • With 20 proposals, average mAP rises from 41.93 at 1 refinement step to 43.72 at 5 steps.Metrics increase with iterations for both tasks, stricter metrics show more pronounced gains, and more iterations cost more time; IoU also improves across denoising layers.Source: Table VI · p. 10, C. Ablation Studies · p. 10, Figure 5 · p. 10

Effectiveness of the Module Components

  • Removing moment diffusion training causes failure to generate targets from noise, while removing it and substituting learnable queries significantly degrades almost all metrics.The highlight branch is pivotal for highlight detection, and each tested model component contributes improvements overall.Source: C. Ablation Studies · p. 10, Table VII · p. 11

Effectiveness of the Loss Components

  • Removing either span loss L1 or Liou lowers performance, especially Liou, while omitting saliency losses harms highlight detection and also affects retrieval.Lclass addresses the imbalance between timestamp positives and the many video frames by distinguishing foreground spans from background padding.Source: C. Ablation Studies · p. 10, Table VIII · p. 11

Effect of Matching Strategy

  • Incrementally increasing training proposal spans from 1 significantly improves performance by reducing early bipartite-matching instability.Instability can occur when training proposal counts substantially exceed the ground-truth count because optimization goals are inconsistent.Source: C. Ablation Studies · p. 11, Table IX · p. 11

Random Seed

  • Across five independently trained instances, moment retrieval averages consistently exceeded 43.4 mAP and highlight detection hovered around 39.6 mAP, with marginal variation.Each instance was evaluated 10 times at the same checkpoint.Source: C. Ablation Studies · p. 11

D. Visualization

The visualization compares DiffusionVMR with Moment-DETR on predicted spans and saliency scores, including examples with multiple targets and similar query-related parts. The conclusion presents DiffusionVMR as a joint denoising framework for retrieval and highlight detection, reporting effectiveness and flexibility across five datasets and three settings.

D. Visualization

  • Figure 7 summarizes results across five independently trained instances with different random seeds, each evaluated 10 times at the same checkpoint.The caption identifies this as a statistical-results visualization.Source: Figure 7 · p. 11
  • Blue intervals and curves show DiffusionVMR spans and saliency scores, while red results show Moment-DETR; examples include multiple targets and similar query-related parts.The text reports more accurate span prediction and stronger highlight detection for DiffusionVMR in the illustrated comparisons.Source: D. Visualization · p. 11, Figure 8 · p. 12

V. CONCLUSION

  • DiffusionVMR unifies moment retrieval and highlight detection as conditional denoising; the conclusion reports effectiveness and flexibility across five datasets and three settings.Retrieval candidates begin from noise and are iteratively refined, while highlight detection generates clip saliency scores through denoising.Source: §V · p. 11, Figure 8 · p. 12

Related papers

Loading 2308.15109v2…