PaperlayerView exact revision on arXiv

Source-linked AI summary

X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval

Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, Rongrong Ji

arXiv:2207.07285v2cs.CV

TL;DR

Video-text retrieval needs to handle coarse and fine-grained interactions while filtering redundant frames and words. X-CLIP introduces cross-grained contrast and AOSM similarity aggregation, achieving stronger performance than prior methods across five benchmarks.

  • Problem

    Prior video-text retrieval methods primarily use coarse-grained or fine-grained contrast, leaving cross-grained contrast and effective similarity aggregation underexplored.

  • Method

    X-CLIP performs video-sentence, video-word, sentence-frame, and frame-word contrasts, then uses AOSM attention over similarity vectors and matrices to weight relevant frames and words.

  • Results

    X-CLIP outperforms previous state-of-the-art methods on five video-text retrieval datasets, including 49.3 R@1 on MSR-VTT and 50.4 R@1 on MSVD.

  • Takeaways & Limitations

    Multi-grained contrast and AOSM reduce the negative effects of unnecessary frames and unimportant words during retrieval.

Abstract

from arXiv · show

Video-text retrieval has been a crucial and fundamental task in multi-modal research. The development of video-text retrieval has been considerably promoted by large-scale multi-modal contrastive pre-training, which primarily focuses on coarse-grained or fine-grained contrast. However, cross-grained contrast, which is the contrast between coarse-grained representations and fine-grained representations, has rarely been explored in prior research. Compared with fine-grained or coarse-grained contrasts, cross-grained contrast calculate the correlation between coarse-grained features and each fine-grained feature, and is able to filter out the unnecessary fine-grained features guided by the coarse-grained feature during similarity calculation, thus improving the accuracy of retrieval. To this end, this paper presents a novel multi-grained contrastive model, namely X-CLIP, for video-text retrieval. However, another challenge lies in the similarity aggregation problem, which aims to aggregate fine-grained and cross-grained similarity matrices to instance-level similarity. To address this challenge, we propose the Attention Over Similarity Matrix (AOSM) module to make the model focus on the contrast between essential frames and words, thus lowering the impact of unnecessary frames and words on retrieval results. With multi-grained contrast and the proposed AOSM module, X-CLIP achieves outstanding performance on five widely-used video-text retrieval datasets, including MSR-VTT (49.3 R@1), MSVD (50.4 R@1), LSMDC (26.1 R@1), DiDeMo (47.8 R@1) and ActivityNet (46.2 R@1). It outperforms the previous state-of-theart by +6.3%, +6.6%, +11.1%, +6.7%, +3.8% relative improvements on these benchmarks, demonstrating the superiority of multi-grained contrast and AOSM.

1 INTRODUCTION

X-CLIP addresses limitations of coarse- and fine-grained video-text contrast by introducing cross-grained contrast and AOSM for filtering and aggregating relevant frame-word interactions. It reports improved retrieval across five benchmarks.

  • 1 INTRODUCTION: Video-text retrieval seeks the most relevant video or text for a text or video query, but whole-sentence and whole-video embeddings miss fine-grained interactions.Recent contrastive pre-training has advanced retrieval, while CLIP and CLIP4Clip primarily represent complete sentences and videos.
  • 1 INTRODUCTION: Cross-grained contrast compares coarse-grained features with each fine-grained feature to filter unimportant frames and words during retrieval.It complements coarse-grained video-sentence and fine-grained frame-word contrasts by using video-word and sentence-frame relationships.
  • 1 INTRODUCTION: X-CLIP combines video-sentence, video-word, sentence-frame, and frame-word contrasts with AOSM attention over similarity vectors and matrices.AOSM dynamically weights frames and words rather than using conventional Mean-Max aggregation, reducing the influence of unnecessary content.
  • 1 INTRODUCTION: 49.3 R@1 is achieved on MSR-VTT, representing a 6.3% relative and 2.9% absolute improvement over the previous state of the art.This result is reported as part of experiments on five widely used video-text retrieval benchmarks.
  • 1 INTRODUCTION: 50.4 R@1, 26.1 R@1, 47.8 R@1, and 46.2 R@1 are achieved on MSVD, LSMDC, DiDeMo, and ActivityNet, respectively.The reported relative improvements over previous state of the art are +6.6%, +11.1%, +6.7%, and +3.8%, respectively.

2 RELATED WORKS

Related work progresses from task-specific video-text fusion and end-to-end retrieval toward contrastive methods, while prior approaches largely emphasize coarse- or fine-grained alignment. X-CLIP targets the less explored cross-grained setting.

  • 2 RELATED WORKS: CLIP, ALIGN, and related contrastive pre-training methods improve multimodal retrieval by learning representations from large-scale image-text data.FILIP extends contrastive learning to patch-word alignment, while TACo combines token- and sentence-level losses.
  • 2 RELATED WORKS: Traditional video-text retrieval uses task-specific or modality-specific fusion over pre-extracted features, whereas newer methods train end-to-end from raw video and text.Examples include MIL-NCE, ClipBERT, Frozen in Time, and CLIP4Clip.
  • 2 RELATED WORKS: Previous end-to-end video-text work investigates video-sentence similarity, but cross-grained video-word and sentence-frame contrast has been rarely explored.X-CLIP considers video-sentence, video-word, sentence-frame, and frame-word contrasts together.

3 METHODOLOGY

The methodology extracts multi-grained visual and textual representations, learns multi-grained contrastive scores and matrices, and aggregates them into instance-level similarity.

  • 3 METHODOLOGY: X-CLIP first extracts multi-grained visual and textual representations, then performs multi-grained contrastive learning and aggregates similarity vectors or matrices.The architecture includes representation extraction, contrastive score computation, and instance-level similarity aggregation.

3.1 Feature Representation

X-CLIP builds frame- and video-level visual features with temporal modeling and averages, while CLIP encoders provide textual representations and word-level features.

  • 3.1 Feature Representation: Frames are sampled at 1 FPS and processed by a 12-layer ViT initialized from public CLIP checkpoints to obtain frame-level features.The final-layer [CLS] tokens serve as frame representations.
  • 3.1.1 Frame-level Representation: A three-layer temporal Transformer models interactions among separately extracted frame features using predefined temporal position embeddings.This adds temporal relationships that independent frame encodings do not capture.
  • 3.1 Feature Representation: The final visual representation contains frame-level features for all n frames, from which a video-level feature is obtained by averaging.The passage defines V_i as the sequence of final fine-grained frame features and n as the frame count.
  • 3.1 Feature Representation: The CLIP text encoder generates textual representations, while final-layer [EOS] and word-token outputs provide sentence- and word-level features.The word-level feature sequence contains m tokens, where m is the sentence length.

3.2 Multi-Grained Contrastive Learning

X-CLIP extends video-text retrieval beyond coarse- and fine-grained contrast by adding cross-grained similarities between coarse representations and individual words or frames. Its encoders produce multi-grained visual and textual features used to calculate four similarity types.

  • Multi-grained contrast: X-CLIP contrasts video-sentence, video-word, sentence-frame, and frame-word representations for multi-grained retrieval.The framework explicitly combines coarse-grained, fine-grained, and cross-grained comparisons.
  • Cross-grained contrast: Cross-grained contrast compares coarse-grained features with each fine-grained feature to filter unnecessary information.Video-word and sentence-frame contrasts provide the cross-grained comparisons.
  • Cross-grained similarity: Video-level representations are compared with every word, and sentence-level representations with every frame, using matrix multiplication.These operations produce similarity vectors for video-word and sentence-frame contrast.
  • Representation extraction: The text encoder generates coarse- and fine-grained textual representations, while sampled video frames pass through frame and temporal encoders to produce visual representations.The coarse-grained visual representation is obtained by averaging fine-grained visual features.
  • Fine-grained similarity: The fine-grained similarity matrix contains pairwise similarities between the n video frames and m sentence words.Matrix multiplication obtains the frame-word similarity matrix.

3.3 Attention Over Similarity Matrix (AOSM)

AOSM aggregates fine-grained similarity vectors and matrices into instance-level scores while assigning different weights to similarity values. Its attention operations emphasize query-relevant frames and words during aggregation.

  • AOSM motivation: AOSM assigns different weights to similarity-vector and similarity-matrix scores when producing instance-level similarity.It addresses the limitation that Mean-Max aggregation ignores differing frame and word importance.
  • Vector aggregation: Softmax gives higher weights to fine-grained features related to the query before similarity scores are aggregated.The temperature parameter τ controls the Softmax.
  • Matrix aggregation: AOSM applies attention twice to the frame-word similarity matrix to obtain video-level and sentence-level similarity vectors.The matrix contains similarities for n frames and m words.
  • Similarity vectors: The video-level vector records similarities between the video and sentence words, while the sentence-level vector records similarities between the sentence and video frames.These vectors are then used for fine-grained instance-level similarity aggregation.
  • Score formation: The fine-grained similarity score is obtained using the average value of the relevant similarity representation.The passage identifies averaging as the fine-grained similarity-score calculation.

3.4 Similarity Calculation

X-CLIP defines instance similarity as a combination of multi-grained contrastive similarity scores rather than relying only on coarse-grained contrast. The resulting score measures semantic similarity between a video and a sentence.

  • Final similarity: The final similarity score s(v_i,t_j) contains multi-grained contrastive similarity scores.This differs from prior work described as considering only coarse-grained contrast.

3.5 Objective Function

During training, X-CLIP forms a batch-level video-text similarity matrix and optimizes it with a symmetric InfoNCE loss. The objective is defined over the similarities among B video-text pairs.

  • Batch objective: For a batch of B video-text pairs, X-CLIP generates a B × B similarity matrix.Each matrix entry represents a video-text similarity used during training.
  • Contrastive loss: The retrieval model is optimized with symmetric InfoNCE loss over the batch similarity matrix.The loss is applied symmetrically to the video-to-text and text-to-video directions.

4 EXPERIMENTS

Experiments evaluate X-CLIP across five video-text retrieval datasets, compare multi-grained components and aggregation strategies, and examine qualitative retrieval behavior. X-CLIP achieves state-of-the-art results, while ablations support the contributions of cross-grained contrast, AOSM, and temporal modeling.

  • Performance Comparison: X-CLIP achieves state-of-the-art results on MSR-VTT, MSVD, LSMDC, DiDeMo, and ActivityNet.
  • Performance Comparison: 49.3 R@1 on MSR-VTT text-to-video retrieval represents a 6.3% relative and 2.9% absolute improvement over CLIP4Clip-seqTransf.
  • Performance Comparison: On MSVD, LSMDC, DiDeMo, and ActivityNet, X-CLIP improves text-to-video R@1 over CLIP4Clip with ViT-B/16 by 6.6%, 11.1%, 6.7%, and 3.8% relatively.The corresponding absolute improvements are 3.1%, 2.6%, 3.0%, and 1.7%.
  • Ablation Study: Adding cross-grained contrast modules yields 1.0% and 0.7% absolute R@1 improvements without conflicting with coarse- and fine-grained contrast.
  • Ablation Study: AOSM outperforms Mean-Max, Max-Max, Max-Mean, and Mean-Mean similarity aggregation strategies.Mean-Mean applies equal weights and cannot eliminate adverse effects from unnecessary frames and unimportant words; top-1 strategies can ignore important scores.
  • Ablation Study: X-CLIP with a temporal encoder consistently outperforms versions without one for both ViT-B/32 and ViT/16.The temporal encoder models relations among frames and supports information requiring multiple frames, such as actions.

5 CONCLUSION

X-CLIP is presented as an end-to-end multi-grained contrastive model for video-text retrieval. Its multi-grained contrasts and AOSM module reduce the effects of unnecessary frames and words, with gains reported on five datasets.

  • X-CLIP encodes sentences and videos into coarse-grained and fine-grained representations for multi-grained contrastive learning.
  • Figure 4 presents top-3 text-to-video retrieval results on MSR-VTT, with similarity scores shown in parentheses.
  • The model combines fine-grained, coarse-grained, and cross-grained contrasts to reduce the negative effects of unnecessary frames and unimportant words during retrieval.
  • Significant performance gains are reported on five popular video-text retrieval datasets.

6 APPENDIX

The appendix compares X-CLIP with CLIP4Clip variants, examines individual contrastive modules, and evaluates AOSM against Transformer modeling. These analyses report broad variant comparisons and attribute AOSM’s advantage to explicit importance calculation, fewer parameters, and dot-product similarity.

  • Variant comparisons: X-CLIP outperforms all compared CLIP4Clip variants across the evaluated backbones and retrieval benchmarks.
  • Contrastive-module analysis: When training data is sufficient, the four single-contrast variants show similar video-to-text and text-to-video retrieval performance.
  • AOSM versus Transformer modeling: AOSM performs better than a 3-layer Transformer for modeling multi-grained features.
  • AOSM versus Transformer modeling: AOSM explicitly calculates frame and word importance, contains fewer parameters, and is described as easier to optimize than the Transformer alternative.
  • AOSM versus Transformer modeling: AOSM obtains similarity scores by dot product, retaining CLIP’s prior knowledge, whereas the Transformer alternative uses a Linear layer.
  • Benchmark comparisons: Tables 10–13 report retrieval-performance comparisons on MSVD, LSMDC, DiDeMo, and ActivityNet.
Loading 2207.07285v2…