PaperlayerView exact revision on arXiv

Source-linked AI summary

MomentDiff: Generative Video Moment Retrieval from Random to Real

Pandeng Li, Chen-Wei Xie, Hongtao Xie, Liming Zhao, Lei Zhang, Yun Zheng, Deli Zhao, Yongdong Zhang

arXiv:2307.02869v2cs.CV

TL;DR

Video moment retrieval seeks temporal segments matching language descriptions, while location biases in dataset annotations can limit generalization. MomentDiff iteratively denoises random spans using text-video similarity. The authors report efficiency and transferability across public and anti-bias datasets, and greater robustness under location shifts.

  • Problem

    Video moment retrieval seeks moments in a video that match a language query, but models may fit annotation location biases and generalize poorly under location shifts.

    Source: §1 · pp. 1–2

  • Method

    MomentDiff uses diffusion to iteratively refine random temporal spans toward real moments, guided by text-video similarity.

    Source: §1 · p. 2

  • Results

    MomentDiff is reported more efficient and transferable than state-of-the-art methods on three public datasets and two anti-bias datasets.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Charades-STA-Len and Charades-STA-Mom provide location-distribution-shift datasets for evaluating temporal location bias.

    Source: §1 · p. 3

    Diffusion requires multiple iterations, which may affect inference speed; reducing iterations sacrifices some performance, so step count requires a speed-performance trade-off.

    Source: §5 · p. 10

Abstract

from arXiv · show

Video moment retrieval pursues an efficient and generalized solution to identify the specific temporal segments within an untrimmed video that correspond to a given language description. To achieve this goal, we provide a generative diffusion-based framework called MomentDiff, which simulates a typical human retrieval process from random browsing to gradual localization. Specifically, we first diffuse the real span to random noise, and learn to denoise the random noise to the original span with the guidance of similarity between text and video. This allows the model to learn a mapping from arbitrary random locations to real moments, enabling the ability to locate segments from random initialization. Once trained, MomentDiff could sample random temporal segments as initial guesses and iteratively refine them to generate an accurate temporal boundary. Different from discriminative works (e.g., based on learnable proposals or queries), MomentDiff with random initialized spans could resist the temporal location biases from datasets. To evaluate the influence of the temporal location biases, we propose two anti-bias datasets with location distribution shifts, named Charades-STA-Len and Charades-STA-Mom. The experimental results demonstrate that our efficient framework consistently outperforms state-of-the-art methods on three public benchmarks, and exhibits better generalization and robustness on the proposed anti-bias datasets. The code, model, and anti-bias evaluation datasets are available at https://github.com/IMCCretrieval/MomentDiff.

1 Introduction

Video moment retrieval seeks temporal boundaries in untrimmed videos that semantically match language queries. MomentDiff replaces dataset-fitted proposals with random spans refined by text-video similarity, addressing concerns about location bias.

1 Introduction

  • VMR identifies the start and end times in a video that best semantically correspond to a language query.The introduction contrasts this with video classification and action detection, which predict predefined action categories.Source: §1 · p. 1
  • Learnable proposals or queries can fit frequent training locations, potentially overlooking out-of-sample events and limiting generalization under location shifts.The introduction contrasts this with random initialization, which does not rely on distribution-specific proposals or queries.Source: §1 · p. 2
  • MomentDiff iteratively refines random spans with a diffusion denoiser guided by similarity relations between the text query and video frames.Its denoiser maps randomly initialized spans into the multimodal space and refines them toward real spans.Source: §1 · pp. 2–3
  • The authors introduce Charades-STA-Len and Charades-STA-Mom, anti-bias datasets with location distribution shifts for evaluating temporal location bias.They report that MomentDiff is more efficient and transferable than state-of-the-art methods on three public datasets and two anti-bias datasets.Source: §1 · p. 3

2 Related Work

Related work covers proposal-based approaches to video moment retrieval and the emerging use of diffusion models for video-text perception. The paper positions MomentDiff as using similarity-aware multimodal cues to generate moment boundaries from noise.

Video Moment Retrieval

  • VMR methods include predefined proposals, proposal-free boundary regression, and learnable proposals or queries [37].Predefined dense proposals can introduce redundant computation and many negative samples.Source: §2 · p. 3
  • Proposal-learnable methods can outperform proposal-free methods, but fitting target-location distributions may create location bias and limit generalization.The paper says its random inputs make no assumptions about location to alleviate this problem.Source: §2 · p. 3

Diffusion models

  • Diffusion models add noise to data through a Markov chain and learn the reverse process to generate samples; MomentDiff uses multimodal similarity cues to guide boundary generation.The paper describes diffusion models as less explored for video-text perception and identifies its work as the first adaptation to video moment retrieval.Source: §2 · p. 3

3 Method

MomentDiff frames video moment retrieval as predicting a text-relevant span’s center and duration, then progressively denoising random spans using text-video similarity.

3.2 The MomentDiff Framework

  • MomentDiff combines a Similarity-aware Condition Generator with a Video Moment Denoiser that progressively produces temporal targets from random noise.The generator fuses text and video features; the denoiser uses those embeddings to guide span generation.Source: §3.2 · p. 4, Figure 2 · p. 4

3.2.1 Visual and Textual Representations.

  • The framework uses VGG, C3D, or CLIP visual features, adds SlowFast features to CLIP for temporal information, and tests GloVe or CLIP text features.PANN audio features are also explored.Source: §3.2.1 · p. 4

3.2.2 Similarity-aware Condition Generator

  • The Similarity-aware Condition Generator maps text and video features into a shared space, applies cross-attention, and uses self-attention to form fusion embeddings.Its similarity loss combines pointwise cross-entropy and pairwise margin loss to retain coarse-grained similarity semantics for denoising.Source: §3.2.2 · pp. 4–5

3.2.3 Video Moment Denoiser

  • The Video Moment Denoiser learns to reverse noise on spans, using random locations to mitigate reliance on dataset location bias.It predicts spans from the conditioned denoiser network rather than the traditional diffusion model’s predicted mean.Source: §3.2.3 · p. 5
  • The denoiser projects spans into an embedding space and adds a sinusoidal noise-intensity embedding before intensity-aware attention with fusion features.Spans are normalized to [0, 1] and rescaled to [−λ, λ] for the forward process, then clamped and mapped back.Source: §3.2.3 · p. 5
  • Training minimizes span L1 and generalized IoU losses plus confidence cross-entropy, matching predicted and ground-truth spans by Hungarian cost when multiple targets exist.Extra input spans are random noise; the lowest-cost match determines the loss, and only that prediction receives a positive confidence label.Source: §3.2.3 · p. 6

3.3 Inference

  • At inference, MomentDiff samples Gaussian noise, iteratively refines spans from coarse to fine, and selects the final span with the highest confidence.The final step directly uses the network prediction, without post-processing such as box renewal or self-conditioning.Source: §3.3 · p. 6

4 Experiments

MomentDiff is evaluated on three video moment retrieval benchmarks, under location-distribution shifts, and through ablation and qualitative studies. It reports gains over MomentDETR on the public benchmarks and greater robustness on the anti-bias datasets.

4.1 Datasets, Metrics and Implementation Details

  • The experiments use Charades-STA, QVHighlights, and TACoS, with R1@n, MAP@n, and MAPavg as evaluation metrics.The paper describes these datasets as covering daily activities, diverse video themes, and long-form cooking videos, respectively.Source: §4.1 · p. 6

4.2 Performance Comparisons

  • 2.35%, 3.86%, and 13.13% average gains in R1@0.5 over MomentDETR on Charades-STA, QVHighlights, and TACoS, respectively.The comparison covers the Charades-STA, QVHighlights, and TACoS results.Source: Table 1 · p. 7, Table 2 · p. 7, Table 3 · p. 7
  • MomentDiff exceeds MomentDETR by a large margin on Charades-CD and ActivityNet-CD, and shows greater robustness than MomentDETR on Charades-STA-Len and Charades-STA-Mom.On the anti-bias datasets, the performance gap grows relative to the original Charades-STA comparison; the authors also report predictions closer to the test-set distribution.Source: §4.2 · pp. 7–8, Table 4 · p. 8, Table 5 · p. 8

4.3 Ablation Study

  • Linear projection (FC) outperforms ROI span embedding, while removing the denoiser causes a drastic performance drop and removing noise intensity reduces R1@0.5 by 5.16%.The authors attribute the FC result to its trainable soft projection and decoupling from fusion embeddings; they report that denoising training and explicit noise-intensity aggregation improve noise modeling.Source: §4.3 · p. 9
  • Pointwise and pairwise similarity constraints support denoising conditions, while the VMR loss supports localization; the paper reports these training procedures as complementary.Five random spans are reported to achieve good results; the authors caution that more random inputs may make training difficult and degrade performance.Source: §4.3 · p. 9
  • Two diffusion steps provide good results and fast inference, while further iterations can improve high-IoU performance, including R1@0.7.The qualitative examples show predictions from multiple random initial locations converging close to the ground truth.Source: Figure 5 · p. 9, §4.3 · p. 10, §4.4 · p. 10

5 Limitation and Conclusion

MomentDiff uses diffusion to refine predictions from random noise and is reported to alleviate location bias while generalizing across diverse and anti-bias datasets. Its multiple inference iterations create a speed trade-off that can be adjusted by reducing the step count.

Limitation

  • Multiple diffusion iterations may affect inference speed compared with existing methods.Source: §5 · p. 10
  • Reducing the iteration count incurs only a small performance sacrifice, so the paper recommends choosing a step number to balance performance and speed.Source: §5 · p. 10

Conclusion

  • MomentDiff's denoising process refines predictions from random noise and alleviates location bias, with efficiency and generalization reported across diverse and anti-bias datasets.Source: §5 · p. 10

A.1 Datasets

This appendix describes the public benchmark datasets and the shifted location distributions used to construct two anti-bias datasets. It also presents MomentDiff's training pseudocode and Gaussian-initialized iterative inference.

A.1 Datasets

  • MomentDiff is evaluated on Charades-STA, QVHighlights, TACoS, and ActivityNet-Captions, each with its specified video-query pairs, annotations, and data splits.Source: §A.1 · p. 15
  • Charades-STA-Len shifts the training and test distributions from 80% short spans to 80% long spans, while Charades-STA-Mom shifts the distributions across temporal positions.Source: §A.1 · p. 15, Table 7 · p. 15
  • The training pseudocode forms text-video fusion features, noises normalized spans, predicts spans and confidence scores, and combines similarity and retrieval losses.Source: §A.1 · p. 16

A.2 Pseudo Code of MomentDiff

  • Inference begins with spans sampled from a Gaussian distribution and progressively refines them through denoising sampling.Source: §A.2 · p. 16

B Inference Efficiency of MomentDiff

On Charades-STA, MomentDiff's reported inference comparisons show faster performance than 2DTAN and MMN at Step=1, and better results than MomentDETR at similar inference time at Step=10. Its inference procedure iteratively updates sampled spans.

B Inference Efficiency of MomentDiff

C.1 Experiment on ActivityNet-Captions

On ActivityNet-Captions, MomentDiff is reported to improve over MomentDETR and remain competitive with state-of-the-art methods. It also compares favorably with MMN in training and testing speed.

C.1 Experiment on ActivityNet-Captions

  • MomentDiff is 25 times faster than MMN in training time and 7.24 times faster in testing time.Source: §C.1 · p. 17, Table 9 · p. 18
  • The experiment records one-epoch training time and inference time for all test samples on a single NVIDIA Tesla A100 GPU.Source: §C.1 · p. 17
  • On ActivityNet-Captions, MomentDiff improves over the MomentDETR baseline and achieves competitive results compared with state-of-the-art methods.Source: §C.1 · p. 17, Table 9 · p. 18

C.2 Error bars

MomentDiff converges to stable results across the tested random seeds and feature configurations, supporting its ability to generate real spans from arbitrary random spans.

C.2 Error bars

  • Across five random seeds, MomentDiff consistently converges and achieves stable results.The tested seeds were 2023, 2022, 2021, 2020, and 2019.Source: §C.2 · p. 18
  • The stability finding holds across the tested VGG/Glove, C3D/Glove, and SF+C feature configurations.Source: §C.2 · p. 18
  • The authors interpret this stability as evidence that the model can generate real spans from arbitrary random spans.Source: §C.2 · p. 18

C.3 Ablation study

The ablations identify default choices for sampling, the noise schedule, post-processing, batch size, architecture, and loss weighting, while testing span count. The broader-impact discussion notes potential misuse when queries contain violent or pornographic terms.

C.3 Ablation study

  • DDIM and DDPM perform similarly, but DDIM samples faster, so it is the default.Source: §C.3 · p. 18, Table 10 · p. 19
  • The cosine β schedule works better in this experiment and is set as the default.It slows noisy-span changes at the beginning and end of diffusion, which the authors associate with more stable generation.Source: §C.3 · p. 18, Table 10 · p. 19
  • Box Renewal slightly improves results, but is omitted by default to keep inference simple.Source: §C.3 · p. 18, Table 10 · p. 19
  • Batch size 32 achieves the best results among the tested settings.Source: §C.3 · p. 18, Table 10 · p. 19
  • The default (2+2:2) layer setting works best: SCG uses two cross-attention and two self-attention layers, while VMD uses two cross-attention layers.Source: §C.3 · p. 18, Table 10 · p. 19
  • With Lvmr fixed at 1, the default Lsim weight of 4 works best in the tested weight influence experiment.Source: §C.3 · p. 18, Table 10 · p. 19
  • Testing with more random noise boxes slightly improves R1@0.5; training and inference are decoupled, allowing any number of random noises as input.Source: §C.3 · p. 18, Table 10 · p. 19

D Broader Impacts

  • The authors warn that queries containing violent or pornographic terms could enable negative impacts, and plan to consider such scenarios in a more secure VMR model.Source: D Broader Impacts · p. 19

Related papers

Loading 2307.02869v2…