Source-linked AI summary
Multi-modal Transformer for Video Retrieval
Valentin Gabeur, Chen Sun, Karteek Alahari, Cordelia Schmid
TL;DR
Video retrieval needs representations that exploit cross-modal cues and temporal information more fully than existing approaches. The paper introduces a multimodal Transformer with jointly optimized language encoding, and reports state-of-the-art retrieval results on three datasets. Its supported scope includes a model-complexity study deferred to supplementary material.
Problem
Existing video-retrieval methods do not fully exploit cross-modal cues and often aggregate visual features with limited temporal information.
Method
A multimodal Transformer jointly processes features from multiple video modalities across time, while the framework investigates language encoders including BERT for cross-modal similarity estimation.
Results
The framework achieves state-of-the-art video-retrieval results on MSRVTT, ActivityNet, and LSMDC and wins the CVPR 2020 Video Pentathlon Challenge.
Takeaways & Limitations
Jointly contextualizing modality features with Transformer self-attention provides the paper’s supported approach to modeling cross-modal and temporal cues for video retrieval.
Takeaways & Limitations
The paper defers its study of model complexity to the supplementary material.
Abstract
from arXiv · showhide
The task of retrieving video content relevant to natural language queries plays a critical role in effectively handling internet-scale datasets. Most of the existing methods for this caption-to-video retrieval problem do not fully exploit cross-modal cues present in video. Furthermore, they aggregate per-frame visual features with limited or no temporal information. In this paper, we present a multi-modal transformer to jointly encode the different modalities in video, which allows each of them to attend to the others. The transformer architecture is also leveraged to encode and model the temporal information. On the natural language side, we investigate the best practices to jointly optimize the language embedding together with the multi-modal transformer. This novel framework allows us to establish state-of-the-art results for video retrieval on three datasets. More details are available at http://thoth.inrialpes.fr/research/MMT.
1 Introduction
Video retrieval requires representations that capture both cross-modal semantics and temporal structure. The paper introduces a multimodal Transformer framework that jointly processes video modalities and investigates language encoding for retrieval.
- Video retrieval is motivated by the need to access relevant content within billions of hours of online video.
- Effective video representations must combine appearance, motion, audio, speech, and other modalities because their joint cues can provide higher-level semantics.
- The framework jointly uses an MMT video encoder and BERT-based text encoding to estimate cross-modal similarity.
- Prior methods often ignore multimodal signals, process modalities separately, or aggregate frame descriptors while discarding long-term temporal information.
- The proposed multimodal Transformer processes features from different modalities and moments, using self-attention to gather cross-modal and temporal cues.
- The authors investigate language-embedding architectures and report state-of-the-art video-retrieval results on MSRVTT, ActivityNet, and LSMDC.
2 Related work
Related work spans language and video representation learning and visual-language retrieval. The paper distinguishes its approach by jointly encoding multimodal video semantics and temporal dependencies rather than relying only on visual signals or fixed alignments.
- Language representations: Language representation research progressed from bag-of-words and Word2Vec to LSTMs and Transformers that model sequential word relationships through self-attention.
- Video representations: Video representation methods use complementary appearance and motion streams, 3D or separable convolutions, and Transformer-based visual features, but prior work did not study other modalities such as audio.
- Visual-language retrieval: Visual-language retrieval methods embed corresponding audio, image regions, words, or video-caption elements into shared spaces or compare them densely.
- Visual-language retrieval: Some video-retrieval methods assume hierarchical clip-sentence alignment or separate spaces for parts of speech, whereas this work encodes sentences directly without those assumptions.
- Visual-language retrieval: Prior instructional-video work uses HowTo100M but does not fully exploit temporal relations, while this paper uses longer segments to learn temporal dependencies and address speech-visual misalignment.
3 Methodology
The method learns video-caption similarity by encoding multimodal, temporally indexed video features with MMT and combining them with caption embeddings. Caption-conditioned expert weighting and bidirectional max-margin training support retrieval in a shared similarity framework.
- Retrieval framework: MMT ranks videos and captions using a learned similarity function that should score matched video-caption pairs higher than mismatched pairs.Retrieval uses the shared function for both text-to-video and video-to-text ranking.
- Video representation: MMT combines feature semantics, expert identity embeddings, and temporal embeddings, then contextualizes all inputs with stacked self-attention and fully connected layers.Temporal embeddings encode extraction time, aggregated features, and unknown timestamps; self-attention enables cross-modal and temporal interactions.
- Video representation: Each expert extracts semantic feature sequences, which are projected through expert-specific linear layers into the common transformer dimension dmodel.The experts are pretrained models trained for particular tasks and produce features at multiple moments.
- Video representation: The transformer retains one contextualized aggregated output embedding per expert as the video representation.Aggregated expert embeddings are initialized by max pooling corresponding expert features before transformer contextualization.
- Video representation: Compared with collaborative gating, MMT iteratively refines inputs through multiple attention heads and preserves all extracted features with temporal encodings.This design lets each layer attend to all input embeddings and extract event semantics across modalities.
- Caption representation and similarity: The caption encoder uses BERT and projects its [CLS] embedding into expert-specific spaces, while caption-dependent softmax weights combine expert-level similarities.The weighted sum allows different captions to emphasize different video modalities; representations and weights can be precomputed for retrieval.
- Training: Training uses bidirectional max-margin ranking loss to enforce higher similarity for true pairs than for negative video-caption pairs by at least margin m.The loss considers both video-to-caption and caption-to-video mismatches within each batch.
4 Experiments
Experiments evaluate pretraining, language and video encoder choices, expert contributions, and comparisons against prior methods on multiple video-retrieval datasets. Results favor finetuned BERT, multi-modal temporal modeling with all extracted features, and selected expert combinations.
- Pretraining: Pretraining on HowTo100M is evaluated before finetuning on MSRVTT, with additional analyses on ActivityNet and LSMDC.The experiments also examine whether removing stop words helps transfer across datasets.
- Language encoder: Finetuned BERT gives the best caption-representation result, whereas frozen pretrained BERT performs poorly.The authors attribute the frozen model’s weakness to its pretraining objective being different from caption representation.
- Video encoder: MMT outperforms encoder variants without MMT and with Collaborative Gating, while using all extracted features improves contextualization over aggregated inputs.The authors argue that MMT extracts cross-modal information in multiple stages rather than only modulating input embeddings.
- Video encoder: Temporal aggregation can miss multiple events, whereas MMT uses ordered features and can model relative event ordering.The comparison includes ordered and randomly shuffled features.
- Video encoder: Max-pooling expert features initializes the aggregation better than mean pooling or zero vectors, while a 4-layer, 4-head MMT outperforms smaller and larger variants.The model-size comparison contrasts 4×4 with 2×2 and 8×8 configurations.
- Expert comparison: The motion expert performs best alone, while audio contributes most when combined with the other experts; OCR, speech, and face are weakened by missing inputs.The scene expert performs decently alone but adds little alongside other experts.
- Comparisons: The framework is compared on MSRVTT, ActivityNet, and LSMDC, using dataset-specific training, finetuning, and input-length settings.MSRVTT and LSMDC use all seven experts, whereas longer-video datasets use motion and audio experts.
5 Summary
The paper introduces a multi-modal transformer video encoder that attends across modalities and time, and integrates it with a caption encoder for retrieval. The resulting cross-modal framework obtains state-of-the-art video-retrieval results.
- The multi-modal transformer attends to features from different modalities and moments, capturing cross-modal and temporal cues in compact video representations.It is incorporated with a caption encoder in a cross-modal framework for caption-video matching.
- The evaluation covers retrieval performance on MSRVTT, ActivityNet, and LSMDC.Table 4 concerns MSRVTT, while Tables 5 and 6 concern ActivityNet and LSMDC.
- The framework obtains state-of-the-art results for video retrieval.
A.1 Model complexity
The model’s parameter count is dominated by the caption encoder, while adding modalities changes complexity modestly. Feature extraction substantially exceeds the model’s video-processing and ranking times.
- The seven-modality architecture has 133.3M parameters, including 112.9M for caption encoding and 20.4M for video encoding.The video encoder comprises 3.3M projection parameters and 17.1M multi-modal-transformer parameters.
- The two-modality architecture has 127.3M parameters, compared with 133.3M for seven modalities.The multi-modal transformer remains 17.1M parameters in both configurations.
- Replacing the multi-modal transformer with collaborative gating reduces parameters from 133.3M to 123.9M, while inference time changes from 1.1s to 0.8s.The inference-time gain is described as minimal relative to feature extraction.
- Extracting seven experts for 1k MSRVTT videos takes approximately 3000s, versus 1.1s for MMT processing and 0.05s for similarity ranking.Caption processing with BERT and gated embedding modules takes 0.9s.
A.2 Results on additional metrics
Additional evaluations report R@1, R@10, and R@50 for MSRVTT, ActivityNet, and LSMDC. Their results are described as consistent with the main paper’s conclusions.
- The additional evaluations use the retrieval metrics R@1, R@10, and R@50.
- The additional metric evaluations cover MSRVTT, ActivityNet, and LSMDC.They correspond to Tables 7, 8, and 9, respectively.
- The results on these additional metrics are reported to be in line with the main paper’s conclusions.