Source-linked AI summary
HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrieval
Song Liu, Haoqi Fan, Shengsheng Qian, Yiru Chen, Wenkui Ding, Zhongyuan Wang
TL;DR
Video-text retrieval needs better use of transformer feature hierarchies and broader negative interactions than mini-batch training provides. HiT addresses these gaps with hierarchical cross-modal contrastive matching and momentum contrast, and reports advantages across three benchmark datasets.
Problem
Existing cross-modal transformers underuse layer-specific feature characteristics, while end-to-end training restricts negative interactions to mini-batches.
Method
HiT jointly applies feature-level and semantic-level hierarchical matching with momentum-updated key encoders and memory banks for large-scale negative interactions.
Results
HiT reports advantages on the MSRVTT, ActivityNet, and LSMDC video-text retrieval benchmarks.
Takeaways & Limitations
Feature hierarchies and large-scale negative learning are presented as complementary directions for improving cross-modal video-text retrieval.
Takeaways & Limitations
Increasing queue sizes can degrade retrieval performance, probably because some positive samples are misclassified as negatives.
Abstract
from arXiv · showhide
Video-Text Retrieval has been a hot research topic with the growth of multimedia data on the internet. Transformer for video-text learning has attracted increasing attention due to its promising performance. However, existing cross-modal transformer approaches typically suffer from two major limitations: 1) Exploitation of the transformer architecture where different layers have different feature characteristics is limited; 2) End-to-end training mechanism limits negative sample interactions in a mini-batch. In this paper, we propose a novel approach named Hierarchical Transformer (HiT) for video-text retrieval. HiT performs Hierarchical Cross-modal Contrastive Matching in both feature-level and semantic-level, achieving multi-view and comprehensive retrieval results. Moreover, inspired by MoCo, we propose Momentum Cross-modal Contrast for cross-modal learning to enable large-scale negative sample interactions on-the-fly, which contributes to the generation of more precise and discriminative representations. Experimental results on the three major Video-Text Retrieval benchmark datasets demonstrate the advantages of our method.
1. Introduction
HiT targets video-text retrieval by exploiting transformer feature hierarchies and expanding cross-modal negative interactions beyond mini-batches. It combines hierarchical contrastive matching with momentum-based memory banks and key encoders.
- Video-text retrieval faces increasing challenges as internet video content expands.
- Dual-stream architectures reduce cross-modal processing from O(MN) pairwise complexity to O(M + N) unary complexity.
- HiT uses feature-level and semantic-level contrastive matching to exploit different transformer-layer representations for multi-view retrieval.
- Momentum Cross-modal Contrast uses memory banks to enable large-scale negative interactions during cross-modal training.
- Momentum-updated key encoders maintain representation consistency in the memory banks.
- HiT jointly performs hierarchical matching and momentum contrast, with experiments conducted on MSRVTT, ActivityNet, and LSMDC.
2. Related Work
Related work spans classical video-text retrieval, transformer-based cross-modal learning, and contrastive learning mechanisms for enlarging negative-sample sets. HiT builds on hierarchical modeling and momentum-based memory banks.
- Video-Text Retrieval: Prior video-text retrieval methods learn semantic similarity using sequence fusion, multi-level encoding, polysemous embeddings, graphs, or hierarchical reasoning.
- Video-Text Learning with Transformer: BERT and transformer backbones have been adapted for video-text learning through joint representations and interactions between global, local, and temporal signals.
- Video-Text Learning with Transformer: HiT’s pipeline uses transformer-based query and key encoders for video and text, with momentum contrast and hierarchical matching.
- Contrastive Learning: End-to-end contrastive learning uses negatives from the current mini-batch, whereas memory banks provide broader negative-sample learning.
- Contrastive Learning: MoCo improves memory-bank consistency by generating large-scale negative representations with a momentum-updated key encoder.
3. Problem Definition
The task maps videos and English captions into multiple common embedding spaces. Retrieval is based on smaller distances for semantically similar cross-modal samples, with overall similarity determined by hierarchical matching.
- The problem provides M videos, each represented by multiple expert embeddings such as motion, appearance, and audio.
- Captions are represented as English natural-language text.
- Video and text encoders jointly map inputs into L common embedding spaces containing compact cross-modal representations.
- Similar cross-modal samples should have smaller distances than dissimilar samples in the common embedding spaces.
- Overall cross-modal similarity is determined by hierarchical contrastive matching results.
4. Hierarchical Transformer
HiT combines hierarchical cross-modal matching with momentum-based memory banks. It uses lower- and higher-layer video and text representations for feature- and semantic-level contrastive learning, while momentum-updated key encoders support broader negative interactions.
- Architecture: HiT uses separate query and key encoders for both video and text, with the key encoders supporting Momentum Cross-modal Contrast.Removing MCC leaves only the two query encoders.
- Video Encoders: The video input combines pretrained visual features with segment masks, positional embeddings, and expert embeddings.Visual features include motion, appearance, and audio representations.
- Hierarchical Features: HiT extracts lower-layer video and text features for feature-level matching and higher-layer representations for semantic-level matching.Video lower-layer features are pooled and projected; higher-layer features capture more complex semantics, while text uses first- and last-layer features analogously.
- Momentum Cross-modal Contrast: Four memory banks store key text and video representations at feature and semantic levels, with queues updated by enqueuing current representations and removing the oldest batch.These banks provide negative representations for contrastive learning.
- Momentum Cross-modal Contrast: Momentum-updated key encoders maintain representation consistency across memory-bank entries generated in different mini-batches.The momentum coefficient is set to m = 0.999 in the paper.
5. Experiments
HiT is evaluated on three video-text retrieval benchmarks using standard retrieval metrics and multimodal pre-trained features. It outperforms comparison methods, with especially strong MSR-VTT performance.
- Datasets and Evaluation Metrics: Experiments evaluate video-text retrieval on MSR-VTT, ActivityNet Captions, and LSMDC.MSR-VTT includes text-to-video and video-to-text evaluation, while ActivityNet Captions and LSMDC report text-to-video retrieval.
- Datasets and Evaluation Metrics: R@K measures whether a relevant item appears among the top K results, while MedR reports the median rank of correct items.Higher R@K and lower MedR indicate better retrieval; rsum sums all R@K values.
- Implementation Details: HiT uses motion, appearance, and audio expert features, with dataset-dependent numbers of visual features and input words.Motion features come from S3D, audio from VGGish, and appearance from SENet-154; text uses fine-tuned BERT-base-uncased.
- Comparison to State of the Art: 320.3 rsum on MSR-VTT exceeds recent state-of-the-art methods by 19.7.With HowTo100M pretraining, HiT further boosts retrieval performance; it also outperforms comparison methods on ActivityNet Captions and LSMDC.
6. Ablation Study
The ablations examine hierarchical matching, momentum contrast, loss choice, temperature, expert inputs, and feature aggregation. They support two-level matching and momentum encoders while revealing efficiency and queue-size trade-offs.
- Hierarchical Cross-modal Matching: Two-level matching combines feature-level matching from early layers with semantic-level matching from final layers.Additional matching levels can improve retrieval, but n-level matching requires n times retrieval during inference.
- Hierarchical Cross-modal Matching: More matching levels provide clear improvements, but 3-level and 4-level variants lack significant gains, so HiT selects 2-level matching.The choice also supports retrieval efficiency and efficient training with Momentum Cross-modal Contrast.
- Momentum Cross-modal Contrast: Large-scale negatives improve retrieval performance, while overly large queues slightly degrade it, probably because positive samples are misclassified as negatives.The experiments vary video and text queue sizes from 0 to 8,192.
- Momentum Encoders: Removing momentum encoders degrades performance, reaching 48.4% at R@5.Momentum-updated key encoders maintain representation consistency in memory banks.
- Contrastive Loss: InfoNCE performs better than Triplet Ranking Loss in HiT, despite the tuning difficulty involving temperature and batch size.
- Temperature: The best retrieval performance occurs at temperature γ = 0.07, while values of the same magnitude do not obviously change performance.
- Expert Utilization: Motion features perform best alone, audio performs worst alone, and motion plus audio is the best two-expert combination.Audio contributes the most when combined with other experts, indicating complementary cues.
- Feature Aggregation: Average Pooling is compared with Max Pooling, 1D-CNN, and [CLS]-token aggregation for feature aggregation.
7. Conclusion
The paper highlights two contributions: exploiting transformer feature hierarchies for performance gains and using large-scale negative learning in cross-modal retrieval. It also identifies feature fusion and memory-bank operations as directions for improvement.
- Hierarchical Cross-modal Contrastive Matching uses transformer feature hierarchies to achieve performance gains.
- Momentum Cross-modal Contrast enables cross-modal learning to benefit from large-scale negative sample learning.
- Future work includes more effective hierarchical feature fusion and memory-bank operations such as data mixing and hard negative selection.