DiffusionRet: Generative Text-Video Retrieval with Diffusion Model

• Discriminant retrieval models conditional probability p(candidates|query) but omit the underlying p(query), making generalization to unseen data difficult.Source: §1 · pp. 1–2
• DiffusionRet generates the joint probability of candidates and query from noise, while training the generator with generation loss and the feature extractor with contrastive loss.The generator uses Kullback-Leibler divergence, while the feature extractor uses InfoNCE loss.Source: §1 · p. 2
• The authors present DiffusionRet as the first generative text-video retrieval approach and the first adaptation of diffusion models for cross-modal retrieval.Source: §1 · p. 2
• DiffusionRet achieves new state-of-the-art performance on MSRVTT, LSMDC, MSVD, ActivityNet Captions, and DiDeMo, and performs well on out-domain retrieval without modification.Source: §1 · p. 2

• Existing text-video retrieval methods commonly map text and video into a shared latent space and focus on conditional probabilities, leaving out-of-distribution identification challenging.Source: §2 · p. 2
• Diffusion models have been used for image, language, and audio generation, as well as discriminative tasks such as segmentation, grounding, and detection.Source: §2 · pp. 2–3
• DiffusionRet addresses the lack of diffusion-based cross-modal retrieval by generating the joint text-video probability distribution from noise.Source: §2 · p. 3, Figure 2 · p. 3

3.1. Existing Solutions: Discriminant Modeling

• Existing retrieval methods rank candidates using conditional probabilities, without modeling the input distribution, which the paper links to limited generalization on unseen data.The paper describes these methods as similarity learning in a shared text–video embedding space.Source: §3.1 · p. 3

3.2. DiffusionRet: Generation Modeling

• DiffusionRet generates a joint text–video distribution from noise, contrasting with methods that optimize conditional probabilities alone.The generator approximates the data distribution of paired text and video.Source: §3.2 · pp. 3–4

3.2.1 Text-Frame Attention Encoder

• The text-frame attention encoder uses text as the query and video frames as keys and values to produce a text-informed video representation.Text features come from CLIP, while frame features are encoded and aggregated before attention.Source: §3.2.1 · p. 4

3.2.2 Query-Candidate Attention Denoising Network

• Query-candidate attention adds the current noisy distribution to attention weights so candidate evidence informs denoising toward a clean distribution.For video-to-text retrieval, the method swaps the query and candidate modalities in the attention inputs.Source: §3.2.2 · p. 4, Figure 3 · p. 4

3.2.3 Optimization from both Generation Perspective and Discrimination Perspective

• Training combines token-level contrastive alignment of text and video features with a diffusion objective that brings predicted and clean distributions closer.The contrastive loss uses word–frame similarities, while the diffusion loss maximizes the likelihood of paired text and video.Source: §3.2.3 · p. 5
• The forward diffusion process adds Gaussian noise to the ground-truth distribution across noise levels, and the generator predicts the data distribution rather than the noise.The noise schedule is controlled by β_k.Source: §3.2.3 · p. 5

4.1. Experimental Settings

• The evaluation covers five datasets, uses R@K, Rsum, MdR, and MnR, and trains the feature extractor before the generator.The implementation uses CLIP (ViT-B/32) and divides training into a discrimination stage followed by a generation stage.Source: §4.1 · pp. 6–7

4.2. Comparison with State-of-the-art

• DiffusionRet outperforms recent state-of-the-art methods on MSRVTT and achieves consistent improvements on LSMDC, MSVD, ActivityNet Captions, and DiDeMo.The MSRVTT comparison covers both text-to-video and video-to-text retrieval; the other four datasets are reported for text-to-video retrieval.Source: §4.2 · p. 7

4.3. Ablation Study

• KL divergence achieves better retrieval performance than MSE, DDPM and DDIM perform similarly, and the cosine β schedule performs well.The authors select KL divergence, DDIM, and the cosine schedule as defaults.Source: §4.3 · p. 7
• 50 diffusion steps give optimal retrieval performance; hybrid training performs best, and a β scale of 1.0 gives the best performance among tested scales.Pure discriminant training performs better than pure generative training on limited data, while hybrid training achieves the best results; the tested β scale range is [0.1, 2.0].Source: §4.3 · pp. 7–8

4.4. The Efficiency of DiffusionRet

• DiffusionRet is reported as efficient as existing state-of-the-art methods at inference, with a 2.50 M-parameter denoising network.The authors report that feature extraction accounts for about 80% of inference time and that increasing diffusion steps can boost performance at negligible time cost.Source: §4.4 · p. 8

4.5. Out-domain Retrieval

• DiffusionRet performs well in both in-domain and out-of-domain retrieval, while EMCL-Net's out-domain performance is slightly lower than CLIP4Clip's despite its stronger in-domain performance.The authors report that DiffusionRet keeps positive and negative pair similarity distributions more separate than the baselines in both settings.Source: §4.5 · pp. 8–9

4.6. Qualitative Analysis

• The qualitative visualization illustrates progressive recovery of the text-video correlation from noise toward the predicted distribution.Figure 5 shows the process from randomly initialized noise x50 to the final predicted distribution x0, with ground truth highlighted in green.Source: §4.6 · p. 9, Figure 5 · p. 9

4.7. Why Diffusion Models

• The authors argue that diffusion's coarse-to-fine refinement and many-to-many nature suit cross-modal retrieval better than GANs, VAEs, or autoregressive networks.They contrast its progressive correlation discovery with GANs and VAEs, and its many-to-many generation with autoregressive approaches.Source: §4.7 · p. 9

5. Conclusion

• DiffusionRet is presented as the first generative diffusion framework for text-video retrieval, explicitly modeling the joint probability of text and video.The method is optimized from both generation and discrimination perspectives.Source: §5 · p. 9
• The authors report that DiffusionRet is well-applicable to in-domain and out-domain retrieval and is superior to discriminant methods in performance and generalization ability.Source: §5 · p. 9
• The paper describes its work as an initial effort to promote generative methods for text-video retrieval and contribute insights to the community.Source: §5 · p. 9

A. Appendix

• The appendix includes dataset descriptions and implementation details.Source: A. Appendix · p. 12
• It adds video-to-text results for three datasets and further out-domain retrieval experiments and discussion of why diffusion is applied.Source: A. Appendix · p. 12
• Additional visualizations cover the diffusion process, text-frame attention, and text-to-video retrieval examples.Source: A. Appendix · p. 12

A.1.1 Datasets

• MSRVTT contains 10,000 YouTube videos with 20 descriptions each, and evaluation uses the 1K-A testing split with 1K candidates.Source: §A.1.1 · p. 12
• LSMDC contains 118,081 short clips from 202 movies, with 1,000 videos used for testing.Source: §A.1.1 · p. 12
• MSVD contains 1,970 videos, each with approximately 40 descriptions, and uses a 1,200-video training and 670-video test split.Source: §A.1.1 · p. 12
• ActivityNet Captions contains densely annotated segments from 20K YouTube videos; segment descriptions are concatenated into video-paragraphs, with results reported on val1.Source: §A.1.1 · p. 12
• DiDeMo contains 10,464 videos and 40,543 descriptions, with segment descriptions concatenated into video-paragraphs for retrieval.Source: §A.1.1 · p. 12

• The feature extractor uses CLIP (ViT-B/32) with 512-dimensional features and a four-layer temporal transformer with eight heads and 512 hidden channels.Temporal position embeddings and parameters are initialized from the text encoder.Source: §A.1.2 · p. 12
• Table A covers video-to-text results on LSMDC, MSVD, and ActivityNet Captions, while Table B covers out-domain text-to-video results; ↑ and ↓ mark metric direction.Source: §A.1.2 · p. 13
• Short-video datasets use word and frame lengths of 32 and 12, while long-video datasets use lengths of 64 and 64, respectively.MSRVTT, LSMDC, and MSVD are treated as short-video datasets; ActivityNet Captions and DiDeMo as long-video datasets.Source: §A.1.2 · p. 13
• Training first optimizes the feature extractor discriminatively, then optimizes the generator generatively; inference combines representation-space distance with the joint probability of video and text.The candidate sets are assumed to be known in advance for both text-to-video and video-to-text retrieval.Source: §A.1.2 · p. 13

A.2.1 Video-to-Text Retrieval

• DiffusionRet achieves consistent improvements across MSRVTT, LSMDC, MSVD, ActivityNet Captions, and DiDeMo, and handles both short and long videos.Source: §A.2.1 · p. 13

A.2.2 Out-domain Retrieval

• Out-domain evaluation trains on a source dataset and tests on an unseen target; EMCL-Net's in-domain lead over CLIP4Clip reverses slightly, while DiffusionRet performs well in both settings.Source: §A.2.2 · pp. 13–14

A.3.1 Diffusion Process

• The diffusion visualization runs from randomly initialized noise at x50 to a final predicted distribution at x0, illustrating progressive recovery of text-video correlation.Source: §A.2.2 · p. 14, §A.3.1 · pp. 15–16

A.2.3 Why Diffusion Models

• Diffusion models' coarse-to-fine refinement suits retrieval better than GANs or VAEs, while their many-to-many nature suits joint-probability generation better than autoregressive networks.Source: §A.2.3 · p. 14

A.2.4 Limitations of our Work

• Hybrid generation and discrimination training can improve performance with limited data; the authors consider pure generative training more promising when data are sufficient.Source: §A.2.4 · p. 15

A.3.2 Text-Frame Attention Map

• The text-frame attention encoder uses text representations as queries and frame representations as keys and values to select text-similar frames for fine-grained video features.The attention-map results are described as demonstrating captured text-frame correlation.Source: §A.2.3 · p. 15, §A.3.2 · p. 16

A.3.3 Text-to-Video Retrieval

• Two MSRVTT text-to-video examples retrieve the ground-truth videos, illustrating the method's text-video correlation mining.Source: Figure C · p. 15, §A.3.3 · p. 16