Source-linked AI summary
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
Antoine Miech, Ivan Laptev, Josef Sivic
TL;DR
Limited availability of large-scale captioned video data motivates learning text-video embeddings from heterogeneous and incomplete sources. The paper proposes Mixture-of-Embedding-Experts, which combines image-caption and video-caption data and handles missing modalities, including optional face descriptors. On video retrieval benchmarks, the authors report significant improvements and outperform previously reported methods on both retrieval directions.
Problem
Large-scale video-caption datasets are difficult to obtain, motivating methods that learn from heterogeneous and partially available data sources.
Method
MEE learns a joint text-video embedding by weighting expert similarities across available modalities and combining image-caption with video-caption training data.
Results
MEE significantly improves video retrieval and outperforms previously reported methods on both text-to-video and video-to-text tasks.
Takeaways & Limitations
The framework supports simultaneous learning from image and video datasets and can incorporate facial descriptors when available.
Abstract
from arXiv · showhide
Joint understanding of video and language is an active research area with many applications. Prior work in this domain typically relies on learning text-video embeddings. One difficulty with this approach, however, is the lack of large-scale annotated video-caption datasets for training. To address this issue, we aim at learning text-video embeddings from heterogeneous data sources. To this end, we propose a Mixture-of-Embedding-Experts (MEE) model with ability to handle missing input modalities during training. As a result, our framework can learn improved text-video embeddings simultaneously from image and video datasets. We also show the generalization of MEE to other input modalities such as face descriptors. We evaluate our method on the task of video retrieval and report results for the MPII Movie Description and MSR-VTT datasets. The proposed MEE model demonstrates significant improvements and outperforms previously reported methods on both text-to-video and video-to-text retrieval tasks. Code is available at: https://github.com/antoine77340/Mixture-of-Embedding-Experts
1 Introduction
Joint text-video embeddings are useful across video-language applications, but scarce captioned video data motivates learning from heterogeneous and partially available sources. MEE addresses this by combining image-caption and video-caption data, handling missing modalities, and improving retrieval performance.
- Motivation: Joint text-video representations map semantically similar video and text samples to nearby points in a shared embedding space.These representations support tasks including video retrieval, captioning, localization, and summarization.
- Motivation: Captioned video datasets remain limited, with available examples ranging from 10K videos in MSR-VTT to 120K in MPII Movie Description.The paper identifies obtaining many videos with text descriptions as especially difficult.
- Proposed approach: MEE learns joint embeddings from heterogeneous data while handling missing video modalities during training.It combines expert contributions from varying modality inputs and learns the model end-to-end.
- Proposed approach: Images are treated as videos without motion and sound, allowing image-caption data to augment video-caption training.This enables concepts appearing in training images, such as “banana,” to contribute even when absent from training videos.
- Extensions: Adding facial descriptors extends MEE beyond appearance, motion, and sound while accommodating videos without detected faces.The resulting face-augmented model produced consistent retrieval improvements in the reported experiments.
- Contributions: The contributions include image-based augmentation, optional face-descriptor integration, and improved performance across standard text-to-video and video-to-text retrieval benchmarks.The paper reports improvements on several video retrieval benchmarks and state-of-the-art performance on LSMDC benchmarks.
2 Related work
Prior work uses joint multimodal representations and multi-stream video features, while missing-data methods address incomplete inputs. MEE combines these directions by learning with heterogeneous video, image, and optionally facial inputs.
- Vision and Language: Joint vision-language embeddings place semantically similar visual and textual samples close together to support retrieval and other multimodal applications.Related applications include text-to-image/video retrieval, image/video-to-text retrieval, captioning, question answering, and grounding.
- Multi-stream video representation: Multi-stream video representations commonly combine appearance, motion, and audio features to improve video understanding tasks.Prior work extensively studies which descriptors to combine and how to fuse them efficiently.
- Learning with missing data: Missing-data methods address inputs that are corrupted, unavailable for recording, or intentionally omitted, often through imputation or reconstruction.The related-work discussion covers default-value imputation, matrix completion, inpainting, and universal multitask models.
- Learning with missing data: Missing modalities are a specific missing-data setting in which models must operate without streams such as appearance, audio, or motion.This capability enables mixing data types such as images and videos with different available modalities.
- Position of MEE: MEE applies multimodal fusion to heterogeneous inputs while explicitly supporting missing modalities during training and testing.Its model computes text-video similarity from multiple descriptor streams rather than requiring every stream for every sample.
3 Mixture of embedding experts for video and text
The MEE model learns a common text-video embedding from heterogeneous, potentially incomplete descriptor streams by assigning caption-dependent weights to separate modality experts. It aggregates and gates text and video inputs, combines expert similarities, and renormalizes available experts when streams are missing.
- 3.1 Model overview and notation: MEE learns a common embedding space for text and video while allowing each video to contain only a subset of descriptor types.Supported streams can include motion, appearance, audio, or facial appearance.
- 3.2 Text representation: Each descriptor stream is temporally aggregated and passed through its own gated embedding module, while word embeddings are aggregated into a sentence representation.Video streams use NetVLAD or max pooling; the text aggregation module uses NetVLAD.
- 3.4 Gated embedding module: The gated embedding transforms features into a new space and context-gates dimensions to model nonlinear interactions and recalibrate activation strengths.The gating weights are sigmoid outputs between 0 and 1.
- 3.5 Estimating text-video similarity with a mixture of embedding experts: For each video stream, MEE computes a separate text-video similarity using stream-specific text and video embeddings, then combines these expert scores.The text embedding can emphasize words relevant to a particular stream, such as facial expressions for facial descriptors.
- 3.5 Estimating text-video similarity with a mixture of embedding experts: Expert weights are predicted from the input sentence, providing a caption-dependent attention mechanism over modalities.Captions about actions may emphasize motion, while captions about people or sounds may emphasize faces or audio.
- 3.5 Estimating text-video similarity with a mixture of embedding experts: When streams are missing, the final similarity uses the remaining experts after renormalizing their weights, and training backpropagates only through available branches.This supports cases such as silent videos or shots without people.
- 3.6 Bi-directional ranking loss: The model is trained with a bi-directional max-margin ranking loss so matched sentence-video pairs score above mismatched pairs for both retrieval directions.The margin is set to m = 0.2 in practice.
4 Experiments
Experiments evaluate MEE on MPII and MSR-VTT retrieval, including ablations and augmentation with captioned COCO images. Results show limited MPII gains but improved performance across MSR-VTT metrics and strong LSMDC benchmark comparisons.
- Experimental setup: Experiments evaluate MEE on text-to-video and video-to-text retrieval tasks using the MPII movie and MSR-VTT datasets.The study includes ablation analyses and comparisons with state-of-the-art methods.
- MPII ablation: MPII ablations show that incorporating heterogeneous data does not significantly improve retrieval performance.
- Augmenting videos with images: Adding COCO images improves performance on both COCO images and MPII videos despite differences in scenes and captions.The single model is trained jointly on the two data sources.
- Augmenting videos with images: MEE gains video-retrieval performance across all reported metrics when MSR-VTT training is augmented with COCO images.COCO samples are injected using a fixed image-to-video sampling rate during each epoch.
- Comparison with state-of-the-art: MEE significantly outperforms reported LSMDC results, including JSFusion, normalized CCA, and the COCO-augmented C+LSTM+SA+FC7 baseline.The comparison covers the LSMDC challenge test sets.
5 Conclusions
The paper concludes that MEE learns text-video embeddings from heterogeneous data while handling missing video modalities during training. It supports joint training with image-caption and video-caption data and optional facial descriptors, and outperforms reported MPII results.
- Contributions: MEE learns text-video embeddings from heterogeneous data sources while handling missing video input modalities during training.
- Heterogeneous data: Images are treated as videos without motion and sound, enabling training from combined image-caption and video-caption datasets.
- Missing modalities: Facial descriptors can be incorporated when faces are present in videos and omitted when videos contain no people.
- Results: MEE outperforms all reported results on the MPII Movie Description dataset.
- Future scope: The framework opens the possibility of learning text-video embeddings from large-scale weakly supervised image and video datasets such as Flickr 100M.