Self-Supervised MultiModal Versatile Networks

• The work learns representations from visual, audio, and language streams available in large collections of unlabelled videos, without manual annotation.Source: §1 · p. 1
• A multimodal versatile network should accept any of the three modalities, respect their differing granularity, support cross-modal comparison, and process videos or static images efficiently.Source: §1 · pp. 1–2
• The proposed design uses modality-specific backbones and embedding graphs so visual and audio representations remain fine-grained while text is semantically coarser.Source: §1 · p. 2
• Evaluation covers action, image, video-text, and audio tasks across HMDB51, UCF101, Kinetics600, PASCAL VOC, ImageNet, YouCook2, MSRVTT, ESC-50, and AudioSet.Source: §1 · p. 2
• The MMV achieves state-of-the-art performance among self-supervised approaches and reduces the gap to supervised state of the art on these benchmarks.Source: §1 · p. 2
• The contributions include comparing modality embedding graphs, introducing deflation for static-image inputs, and demonstrating representations across multiple downstream modalities.Source: §1 · p. 2

• Single-modality self-supervised learning uses annotation-free pretext tasks for vision and exploits temporal structure in videos.Source: §2 · p. 2
• Vision-language methods embed text and images in a shared vector space, enabling semantic similarity and efficient cross-modal search.Source: §2 · p. 2
• Vision-audio methods exploit temporal co-occurrence or predict whether visual and audio signals originate from the same video.Source: §2 · p. 2
• Earlier vision-audio-language approaches include image-audio description datasets and cross-modal networks using image, audio, and text.Source: §2 · p. 3
• Unlike a comparable cross-modal approach relying on curated annotated datasets, this work requires no annotation.Source: §2 · p. 3
• Prior work commonly inflates image networks into video networks, whereas this paper reverses the route by training on videos and enabling still-image inputs.Source: §2 · p. 3

• Language is obtained from video audio using off-the-shelf automatic speech recognition, while the training data remain unlabelled videos.Source: §3 · p. 3
• The model processes RGB-frame sequences, one-dimensional audio samples, and discrete word tokens through modality-specific backbones.Source: §3 · p. 3
• Backbone outputs pass through projection heads into shared spaces where modalities can be compared using dot products.Source: §3 · p. 3

Modality embedding graphs

• The method considers shared, disjoint, and fine-and-coarse modality embedding graphs, each trading off modality specificity against cross-modal navigation.Source: §3.1 · pp. 3–4
• The FAC design compares vision and audio in a fine-grained space, maps them into a lower-dimensional coarse space with text, and permits audio-text comparison through shared visual grounding.Source: §3.1 · p. 4

Multimodal contrastive loss

• Self-supervised contrastive losses align positive streams sampled from the same video location and separate negative streams sampled from different videos.Source: §3.1 · p. 4
• The text objective uses MIL-NCE because narrations correspond less precisely to video content than synchronized audio does.Source: §3.1 · p. 5

Video-to-image network deflation

• Network deflation converts a video model into one that accepts a single image, avoiding repeated-frame static videos and improving efficiency.Source: §3.2 · p. 5

Experimental scope

• The evaluation spans action, image, video-text, and audio tasks using representations pretrained on HowTo100M and/or AudioSet.Benchmarks include UCF101, HMDB51, Kinetics600, PASCAL VOC, ImageNet, YouCook2, MSRVTT, ESC-50, and AudioSet.Source: §4.1 · p. 6

Design explorations

• Vision-and-audio experiments found contrastive loss improved vision downstream tasks by 2% over logistic loss while matching it on audio.Non-linear projection heads helped visual representations, but were not beneficial for language or audio branches.Source: §4.2 · p. 7
• Three-modality training clearly outperformed paired-modality training on visual tasks and produced competitive audio and video-text representations.On HowTo100M, the three-modality model improved UCF101 and HMDB51, matched paired models on ESC-50, won on MSRVTT, and lost on YouCook2.Source: §4.2 · p. 7
• The FAC merging strategy dominated UCF101, HMDB51, and MSRVTT, performed well on ESC-50, and ranked second on YouCook2.Its results support accounting for the differing specificity of visual, audio, and textual modalities.Source: §4.2 · p. 7

Large-scale experiments

• The learned representations matched or exceeded state-of-the-art self-supervised performance across the reported downstream tasks and evaluation modes.On ESC-50, MMV exceeded the best supervised result by 1.7%, while linear classifiers remained competitive with finetuned methods.Source: §4.3 · p. 8
• +9.2% on Kinetics600 was reported for three-modality training over the two-modality comparison.The authors relate this improvement to semantic information in HowTo100M narrations about objects or actions.Source: §4.3 · p. 8

Transfer to image tasks via network deflation

• Deflated networks performed within around 1% of input-inflated video models while being an order of magnitude more efficient on static images.Naive deflation underperformed severely, indicating that deflation training was necessary.Source: §4.4 · p. 8

• Self-supervised training matches or exceeds state-of-the-art performance for action and audio classification on HMDB51, UCF101, Kinetics600, ESC-50, and AudioSet.Source: §5 · p. 9
• The authors identify Kinetics600 and AudioSet as larger-scale downstream tasks for evaluating progress in self-supervised methods.Source: §5 · p. 9
• The network supports zero-shot text-to-video retrieval.Source: §5 · p. 9
• The deflation process trains on videos while obtaining representations usable for still images.Source: §5 · p. 9
• The authors argue that abundant web video offers a natural route for self-supervised transfer and encourage future work to pursue it.Source: §5 · p. 9

• Learning without labels allows training on any available collection of web video data, greatly increasing the potential training scale.Source: §6 · p. 10
• Learned multimodal embeddings can support indexing visual or audio content for web search.Source: §6 · p. 10
• Self-supervised methods may learn and perpetuate biases present in their data.Source: §6 · p. 10
• Deployment should carefully consider consequences arising from potential underlying data biases.Source: §6 · p. 10

• Appendix A details optimization during training and the evaluation setup.Source: Appendix overview · p. 14
• Appendix B specifies the different backbones, projection heads, and losses for the embedding graphs.Source: Appendix overview · p. 14
• Appendix C reports quantitative evaluation of modality-pair design choices summarized in the main paper.Source: Appendix overview · p. 14

• Training samples 16- or 32-frame video subclips, applies scale jittering, resizes the minimum side to 224, and extracts random 200 × 200 crops.Source: §A.1 · p. 14
• Random horizontal flipping and color augmentation complete the listed video preprocessing pipeline.Source: §A.1 · p. 14
• 500K optimization steps use Adam with β1 = 0.9, β2 = 0.999, ϵ = 10−8, initial learning rate 0.002, and 5K-step linear warm-up.Source: §A.1 · p. 14
• A half-period cosine schedule controls learning-rate decay after warm-up.Source: §A.1 · p. 14
• Batch-normalization mean and variance statistics are aggregated across all workers; the authors observed degraded performance without this sharing.Source: §A.1 · p. 14

Video classification

• UCF101 and HMDB51 use frozen-feature linear classification or full-network fine-tuning, with video clips sampled at 10 FPS.Frozen evaluation uses 16 or 32 frames; fine-tuning averages predictions from 10 linearly spaced clips.Source: §A.2 · p. 14
• Kinetics600 evaluation trains a linear classifier on 32-frame clips and reports top-1 accuracy on the validation set.The classifier is trained for 50 epochs with Adam, warmup, and square-root learning-rate decay.Source: §A.2 · pp. 14–15

Audio classification

• ESC-50 and AudioSet use features from the audio backbone’s last convolution before pooling, with 2-second audio inputs.ESC-50 uses an SVM, whereas AudioSet uses a two-layer MLP predicting 527 classes and reports validation mAP.Source: §A.2 · p. 15

Video-text retrieval

• Zero-shot text-to-video retrieval maps text queries and videos into a shared subspace and ranks videos by cosine similarity.Video embeddings average features from 10 linearly spaced clips, using preprocessing close to pretraining.Source: §A.2 · p. 15, Table 4 · p. 15

Image classification

• Deflated networks are evaluated on PASCAL VOC 2007 and ImageNet using linear classifiers after collapsing temporal filters into 2D filters.For TSM, images pass through the backbone without channel shifting.Source: §A.2 · pp. 15–16

Backbones

• Audio, visual, and text backbones extract embeddings specialized to their respective raw input modalities.The resulting embeddings are passed to projection and shared-graph components.Source: B Model architecture and losses details · p. 16

Projection heads

• Projection heads can be linear or nonlinear, with the nonlinear design following SimCLR.Figure 3d specifies the precise architectures used for both head types.Source: B Model architecture and losses details · p. 16, Figure 3 · p. 17

Shared architecture

• The shared architecture connects modalities in a common embedding graph and defines NCE losses between video-audio and video-text pairs.The video-audio and video-text objectives are described separately.Source: B Model architecture and losses details · p. 16

Disjoint architecture

• The disjoint architecture uses a separate embedding graph and defines corresponding NCE losses for video-audio and video-text relationships.Its graph and losses are illustrated separately from the shared architecture.Source: B Model architecture and losses details · p. 17

Fine and Coarse architecture

• The Fine and Coarse architecture uses a dedicated embedding graph with NCE objectives for video-audio and video-text pairs.The architecture is identified as FAC and illustrated in Figure 3c.Source: B Model architecture and losses details · p. 17

Experimental setup

• Experiments train S3D-G on HowTo100M for 500K steps, using 16-frame clips and a batch size of 512 unless otherwise stated.The design exploration reports frozen top-1 accuracy for UCF101, HMDB51, and ESC-50, plus R@10 for retrieval tasks.Source: C Additional design choices… · pp. 17–18

Visual backbone

• TSM ResNet50 variants consistently outperform S3D-G on vision downstream tasks, with TSM ResNet50x2 at least matching TSM ResNet50.These comparisons use the visual-backbone experiments summarized in Table 6.Source: C Additional design choices… · p. 18, Table 6 · p. 18

Losses

• 2%: NCE outperforms logistic loss on vision downstream tasks and is on par with it on audio for Vision+Audio training.The comparison uses linear heads and S3D-G under frozen evaluation.Source: C Additional design choices… · p. 18, Table 7 · p. 18

Projection heads

• Nonlinear projection heads improve visual downstream performance but are not beneficial for the language and audio branches.The result is reported for frozen UCF101 and HMDB51 visual evaluation.Source: C Additional design choices… · p. 18, Table 8 · p. 19

Data augmentation

• 4%: randomly offsetting audio relative to video by up to 0.8s reduces visual-task performance, indicating synchronization is an important training signal.Standard video augmentation usually helps, while more aggressive SpecAugment can harm audio performance.Source: C Additional design choices… · p. 18, Table 9 · p. 19