Source-linked AI summary
Self-Supervised MultiModal Versatile Networks
Jean-Baptiste Alayrac, Adrià Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, Andrew Zisserman
TL;DR
The paper asks how to learn versatile representations from unlabelled videos while preserving the differing granularity of visual, audio, and language modalities. It introduces multimodal embedding graphs, self-supervised cross-modal contrastive learning, and video-to-image network deflation. MMV achieves state-of-the-art self-supervised performance across multiple video, image, audio, and video-text benchmarks, while supporting static-image inputs.
Problem
The goal is to learn from visual, audio, and language streams without manual annotation while preserving modality specificity and enabling cross-modal comparison and video-or-image processing.
Method
MMV uses modality-specific backbones, embedding graphs, and self-supervised contrastive losses, with FAC separating fine-grained visual-audio and coarse-grained text spaces and deflation adapting video networks to images.
Results
MMV achieves state-of-the-art performance among self-supervised approaches across action, image, video-text, and audio benchmarks, reducing the gap to supervised methods.
Takeaways & Limitations
Training on unlabelled video can produce representations applicable across video, image, audio, and video-text tasks, including static-image inputs through deflation.
Takeaways & Limitations
The method relies on automatic speech recognition to obtain language from audio, and learned data representations may reproduce biases present in the data.
Abstract
from arXiv · showhide
Videos are a rich source of multi-modal supervision. In this work, we learn representations using self-supervision by leveraging three modalities naturally present in videos: visual, audio and language streams. To this end, we introduce the notion of a multimodal versatile network -- a network that can ingest multiple modalities and whose representations enable downstream tasks in multiple modalities. In particular, we explore how best to combine the modalities, such that fine-grained representations of the visual and audio modalities can be maintained, whilst also integrating text into a common embedding. Driven by versatility, we also introduce a novel process of deflation, so that the networks can be effortlessly applied to the visual data in the form of video or a static image. We demonstrate how such networks trained on large collections of unlabelled video data can be applied on video, video-text, image and audio tasks. Equipped with these representations, we obtain state-of-the-art performance on multiple challenging benchmarks including UCF101, HMDB51, Kinetics600, AudioSet and ESC-50 when compared to previous self-supervised work. Our models are publicly available.
1 Introduction
The paper learns self-supervised representations from visual, audio, and narrated language streams, aiming for one versatile network that preserves modality specificity while supporting cross-modal comparison and both video and image inputs.
- The work learns representations from visual, audio, and language streams available in large collections of unlabelled videos, without manual annotation.
- A multimodal versatile network should accept any of the three modalities, respect their differing granularity, support cross-modal comparison, and process videos or static images efficiently.
- The proposed design uses modality-specific backbones and embedding graphs so visual and audio representations remain fine-grained while text is semantically coarser.
- Evaluation covers action, image, video-text, and audio tasks across HMDB51, UCF101, Kinetics600, PASCAL VOC, ImageNet, YouCook2, MSRVTT, ESC-50, and AudioSet.
- The MMV achieves state-of-the-art performance among self-supervised approaches and reduces the gap to supervised state of the art on these benchmarks.
- The contributions include comparing modality embedding graphs, introducing deflation for static-image inputs, and demonstrating representations across multiple downstream modalities.
2 Related work
Prior work develops self-supervised representations within individual modalities and aligns visual, audio, and language signals, while this paper distinguishes itself by learning from unannotated video modalities.
- Single-modality self-supervised learning uses annotation-free pretext tasks for vision and exploits temporal structure in videos.
- Vision-language methods embed text and images in a shared vector space, enabling semantic similarity and efficient cross-modal search.
- Vision-audio methods exploit temporal co-occurrence or predict whether visual and audio signals originate from the same video.
- Earlier vision-audio-language approaches include image-audio description datasets and cross-modal networks using image, audio, and text.
- Unlike a comparable cross-modal approach relying on curated annotated datasets, this work requires no annotation.
- Prior work commonly inflates image networks into video networks, whereas this paper reverses the route by training on videos and enabling still-image inputs.
3 Approach
The approach learns modality-specific and shared representations from unlabelled videos using embedding graphs and contrastive alignment, then deflates video networks to process single images.
- Language is obtained from video audio using off-the-shelf automatic speech recognition, while the training data remain unlabelled videos.
- The model processes RGB-frame sequences, one-dimensional audio samples, and discrete word tokens through modality-specific backbones.
- Backbone outputs pass through projection heads into shared spaces where modalities can be compared using dot products.
- Modality embedding graphs: The method considers shared, disjoint, and fine-and-coarse modality embedding graphs, each trading off modality specificity against cross-modal navigation.
- Modality embedding graphs: The FAC design compares vision and audio in a fine-grained space, maps them into a lower-dimensional coarse space with text, and permits audio-text comparison through shared visual grounding.
- Multimodal contrastive loss: Self-supervised contrastive losses align positive streams sampled from the same video location and separate negative streams sampled from different videos.
- Multimodal contrastive loss: The text objective uses MIL-NCE because narrations correspond less precisely to video content than synchronized audio does.
- Video-to-image network deflation: Network deflation converts a video model into one that accepts a single image, avoiding repeated-frame static videos and improving efficiency.
4 Experiments
The experiments evaluate multimodal representations across video, image, audio, and video-text tasks, while testing modality combinations, merging strategies, and transfer from video to images. Three-modality training and the FAC design generally improve downstream performance, while deflation preserves image performance at much lower computational cost.
- Experimental scope: The evaluation spans action, image, video-text, and audio tasks using representations pretrained on HowTo100M and/or AudioSet.Benchmarks include UCF101, HMDB51, Kinetics600, PASCAL VOC, ImageNet, YouCook2, MSRVTT, ESC-50, and AudioSet.
- Design explorations: Vision-and-audio experiments found contrastive loss improved vision downstream tasks by 2% over logistic loss while matching it on audio.Non-linear projection heads helped visual representations, but were not beneficial for language or audio branches.
- Design explorations: Three-modality training clearly outperformed paired-modality training on visual tasks and produced competitive audio and video-text representations.On HowTo100M, the three-modality model improved UCF101 and HMDB51, matched paired models on ESC-50, won on MSRVTT, and lost on YouCook2.
- Design explorations: The FAC merging strategy dominated UCF101, HMDB51, and MSRVTT, performed well on ESC-50, and ranked second on YouCook2.Its results support accounting for the differing specificity of visual, audio, and textual modalities.
- Large-scale experiments: The learned representations matched or exceeded state-of-the-art self-supervised performance across the reported downstream tasks and evaluation modes.On ESC-50, MMV exceeded the best supervised result by 1.7%, while linear classifiers remained competitive with finetuned methods.
- Large-scale experiments: +9.2% on Kinetics600 was reported for three-modality training over the two-modality comparison.The authors relate this improvement to semantic information in HowTo100M narrations about objects or actions.
- Transfer to image tasks via network deflation: Deflated networks performed within around 1% of input-inflated video models while being an order of magnitude more efficient on static images.Naive deflation underperformed severely, indicating that deflation training was necessary.
5 Conclusion
The paper trains versatile vision, audio, and language networks self-supervised and reports state-of-the-art or better performance across five challenging benchmarks. It also supports zero-shot text-to-video retrieval and transfers video-trained representations to still images.
- Self-supervised training matches or exceeds state-of-the-art performance for action and audio classification on HMDB51, UCF101, Kinetics600, ESC-50, and AudioSet.
- The authors identify Kinetics600 and AudioSet as larger-scale downstream tasks for evaluating progress in self-supervised methods.
- The network supports zero-shot text-to-video retrieval.
- The deflation process trains on videos while obtaining representations usable for still images.
- The authors argue that abundant web video offers a natural route for self-supervised transfer and encourage future work to pursue it.
6 Broader impact
Self-supervised video learning can scale to web video and support multimodal content indexing, but deployment must consider biases learned from the data.
- Learning without labels allows training on any available collection of web video data, greatly increasing the potential training scale.
- Learned multimodal embeddings can support indexing visual or audio content for web search.
- Self-supervised methods may learn and perpetuate biases present in their data.
- Deployment should carefully consider consequences arising from potential underlying data biases.
Appendix overview
The appendices provide optimization and evaluation details, architectural and loss specifications, and quantitative results for modality-pair design exploration.
- Appendix A details optimization during training and the evaluation setup.
- Appendix B specifies the different backbones, projection heads, and losses for the embedding graphs.
- Appendix C reports quantitative evaluation of modality-pair design choices summarized in the main paper.
A.1 Training details
Training preprocesses sampled video clips with spatial and temporal augmentations, then optimizes the networks with Adam using warm-up and cosine learning-rate scheduling; batch statistics are shared across workers.
- Training samples 16- or 32-frame video subclips, applies scale jittering, resizes the minimum side to 224, and extracts random 200 × 200 crops.
- Random horizontal flipping and color augmentation complete the listed video preprocessing pipeline.
- 500K optimization steps use Adam with β1 = 0.9, β2 = 0.999, ϵ = 10−8, initial learning rate 0.002, and 5K-step linear warm-up.
- A half-period cosine schedule controls learning-rate decay after warm-up.
- Batch-normalization mean and variance statistics are aggregated across all workers; the authors observed degraded performance without this sharing.
A.2 Downstream tasks details
The paper evaluates learned representations across video, image, audio, and video-text tasks using linear classification, fine-tuning, and zero-shot retrieval protocols.
- Video classification: UCF101 and HMDB51 use frozen-feature linear classification or full-network fine-tuning, with video clips sampled at 10 FPS.Frozen evaluation uses 16 or 32 frames; fine-tuning averages predictions from 10 linearly spaced clips.
- Video classification: Kinetics600 evaluation trains a linear classifier on 32-frame clips and reports top-1 accuracy on the validation set.The classifier is trained for 50 epochs with Adam, warmup, and square-root learning-rate decay.
- Audio classification: ESC-50 and AudioSet use features from the audio backbone’s last convolution before pooling, with 2-second audio inputs.ESC-50 uses an SVM, whereas AudioSet uses a two-layer MLP predicting 527 classes and reports validation mAP.
- Video-text retrieval: Zero-shot text-to-video retrieval maps text queries and videos into a shared subspace and ranks videos by cosine similarity.Video embeddings average features from 10 linearly spaced clips, using preprocessing close to pretraining.
- Image classification: Deflated networks are evaluated on PASCAL VOC 2007 and ImageNet using linear classifiers after collapsing temporal filters into 2D filters.For TSM, images pass through the backbone without channel shifting.
B Model architecture and losses details
The architecture extracts modality-specific embeddings, applies projection heads, and trains shared or disjoint embedding graphs with NCE-based objectives.
- Backbones: Audio, visual, and text backbones extract embeddings specialized to their respective raw input modalities.The resulting embeddings are passed to projection and shared-graph components.
- Projection heads: Projection heads can be linear or nonlinear, with the nonlinear design following SimCLR.Figure 3d specifies the precise architectures used for both head types.
- Shared architecture: The shared architecture connects modalities in a common embedding graph and defines NCE losses between video-audio and video-text pairs.The video-audio and video-text objectives are described separately.
- Disjoint architecture: The disjoint architecture uses a separate embedding graph and defines corresponding NCE losses for video-audio and video-text relationships.Its graph and losses are illustrated separately from the shared architecture.
- Fine and Coarse architecture: The Fine and Coarse architecture uses a dedicated embedding graph with NCE objectives for video-audio and video-text pairs.The architecture is identified as FAC and illustrated in Figure 3c.
C Additional design choices exploration for pairs of modalities
Pairwise modality experiments isolate backbone, loss, projection-head, and augmentation choices before constructing the three-modality network.
- Experimental setup: Experiments train S3D-G on HowTo100M for 500K steps, using 16-frame clips and a batch size of 512 unless otherwise stated.The design exploration reports frozen top-1 accuracy for UCF101, HMDB51, and ESC-50, plus R@10 for retrieval tasks.
- Visual backbone: TSM ResNet50 variants consistently outperform S3D-G on vision downstream tasks, with TSM ResNet50x2 at least matching TSM ResNet50.These comparisons use the visual-backbone experiments summarized in Table 6.
- Losses: 2%: NCE outperforms logistic loss on vision downstream tasks and is on par with it on audio for Vision+Audio training.The comparison uses linear heads and S3D-G under frozen evaluation.
- Projection heads: Nonlinear projection heads improve visual downstream performance but are not beneficial for the language and audio branches.The result is reported for frozen UCF101 and HMDB51 visual evaluation.
- Data augmentation: 4%: randomly offsetting audio relative to video by up to 0.8s reduces visual-task performance, indicating synchronization is an important training signal.Standard video augmentation usually helps, while more aggressive SpecAugment can harm audio performance.