Source-linked AI summary
Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos
Brian Chen, Andrew Rouditchenko, Kevin Duarte, Hilde Kuehne, Samuel Thomas, Angie Boggust, Rameswar Panda, Brian Kingsbury, Rogerio Feris, David Harwath, James Glass, Michael Picheny, Shih-Fu Chang
TL;DR
Learning comparable video, audio, and text representations is difficult because modalities differ and their real-world alignment can be weak. MCN combines instance-level contrastive learning with multimodal semantic clustering in a common embedding space. In zero-shot evaluations, it outperforms retrieval and temporal action localization baselines on multiple datasets.
Problem
Learning a common multimodal space is challenging because modalities differ in source and processing, and their real-world alignment may be weak; instance contrastive learning also treats different-time samples as negatives regardless of semantic similarity.
Source: §1 · p. 2Method
MCN combines cross-modal instance-level contrastive learning with clustering across joint multimodal representations to bring semantically related instances together.
Source: §1 · p. 2Results
In zero-shot settings, MCN outperforms the best text-to-video retrieval baseline by over absolute 3% in recall and the temporal action localization baseline by over 3.1% in recall.
Source: §1 · p. 2Takeaways & Limitations
The learned common-space representations improve state-of-the-art results on multiple downstream tasks without additional training on target datasets.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Multimodal self-supervised learning is getting more and more attention as it allows not only to train large networks without human supervision but also to search and retrieve data across various modalities. In this context, this paper proposes a self-supervised training framework that learns a common multimodal embedding space that, in addition to sharing representations across different modalities, enforces a grouping of semantically similar instances. To this end, we extend the concept of instance-level contrastive learning with a multimodal clustering step in the training pipeline to capture semantic similarities across modalities. The resulting embedding space enables retrieval of samples across all modalities, even from unseen datasets and different domains. To evaluate our approach, we train our model on the HowTo100M dataset and evaluate its zero-shot retrieval capabilities in two challenging domains, namely text-to-video retrieval, and temporal action localization, showing state-of-the-art results on four different datasets.
1. Introduction
The paper addresses the challenge of learning comparable representations across video, audio, and text despite modality differences and weak alignment. MCN combines instance-level contrastive learning with multimodal semantic clustering, and is evaluated on zero-shot retrieval and temporal action localization.
1. Introduction
- The learning problem is to project modality-specific features into a common space where content-similar features are close for direct cross-modal retrieval.Modality features differ in source, sampling, and processing, while audio, video, and narration may be weakly aligned.Source: §1 · p. 2
- MCN combines instance-level contrastive loss with cluster-level semantic consistency, using joint clusters across modalities rather than separate modality clusters.The framework learns cross-modal projection heads for visual, audio, and language inputs from narrated videos.Source: §1 · p. 2
- MCN outperforms the best text-to-video retrieval baseline by over absolute 3% in recall and the temporal action localization baseline by over 3.1% in recall in zero-shot settings.Evaluation uses HowTo100M training and YouCook2, MSR-VTT, CrossTask, and Mining YouTube; the comparison uses features from pretrained backbones.Source: §1 · p. 2
- The paper reports that common-space representations improve state-of-the-art results on multiple downstream tasks without additional training on target datasets.The reported contributions include joint-space learning across video, audio, and text.Source: §1 · p. 2
2. Related Work
Prior work learns multimodal representations through contrastive alignment or clustering, but these approaches leave a gap between instance-level alignment and cross-modal semantic grouping. MCN instead clusters features jointly in a common multimodal space.
2. Related Work
- The paper’s goal differs from approaches using caption pairs, raw speech, or generated captions: it learns a directly comparable joint embedding across three modalities for zero-shot tasks.The related-work passage names prior approaches with markers [3] [18] [19] [33] [36] [41] [48] [53] and [2] [6] [23] [35] [43].Source: §2 · p. 2
- Instance-wise contrastive learning pulls same-instance samples together but can treat semantically similar samples from different times as negatives; MCN adds clustering to learn semantic similarity among multimodal instances.The cited discussion describes MIL-NCE as allowing compensation for video-caption misalignment [37].Source: §2 · p. 3
- XDC and SeLaVi cluster modalities separately and use cross-domain pseudo-labels, whereas MCN creates a common embedding space and clusters jointly across modalities.The figure contrasts separate-space clustering with joint-space multimodal clustering.Source: §2 · p. 3, Figure 2 · p. 3
- Online clustering can jointly learn visual embeddings and cluster assignments, while clustering over large datasets slows training; multimodal information is also important for clustering performance.Prior multimodal methods extend clustering to audio and video but focus on separate domain-specific feature extractors [2] [5].Source: §2 · p. 3
3. Learning to Cluster Multimodal Data
MCN projects video, audio, and text into a shared embedding space, then combines pairwise contrastive alignment with multimodal clustering and reconstruction regularization. Its clustering operates on fused multimodal features and uses semantic centroids to guide each modality.
3. Learning to Cluster Multimodal Data
- MCN maps associated video, audio, and narration representations into a shared space intended to place semantically similar clips close and dissimilar content apart.Video inputs use pre-extracted 2D and 3D features, audio uses log-mel spectrograms, and text uses a sentence-based neural model.Source: §3 · p. 3
- The training objective sums pairwise contrastive loss, clustering loss, and reconstruction loss for regularizing common-space features.The framework illustration shows feature projection, pairwise contrastive alignment, multimodal clustering, joint centroid prediction, and reconstruction regularization.Source: §3 · p. 4, Figure 3 · p. 4
3.1. Contrastive Loss for Learning Joint Spaces
- Pairwise contrastive loss covers video-text, text-audio, and audio-video pairs, maximizing similarity for same-clip representations and minimizing it for imposter pairs.The total MMS loss is the sum of the three modality-pair losses; imposter pairs are sampled from a batch but do not co-occur.Source: §3.1 · p. 4
3.2. Clustering Multimodal Features
- Clustering uses the mean of video, audio, and text embeddings as each multimodal instance feature, then assigns these features to centroids.The fused feature is the average of the three modality embeddings.Source: §3.2 · p. 5
- Unlike pseudo-label-only methods, MCN uses the centroid matrix as a semantic learning target to pull each modality toward its multimodal instance’s nearest centroid and away from other centroids.The method sums the per-modality clustering losses.Source: §3.2 · p. 5
- Reconstruction on the common-space features is used to stabilize feature training during clustering and regularize training.The paper hypothesizes reconstruction can preserve information suppressed by contrastive learning or clustering, such as foreground speech and background chopping sounds.Source: §3.2 · p. 5
4. Experiments
The experiments evaluate fixed-backbone multimodal embeddings on zero-shot retrieval and temporal localization, alongside full-video retrieval, ablations, and qualitative analyses. MCN outperforms prior approaches across the reported retrieval and localization benchmarks.
4.1. Implementation details
- With fixed pretrained backbones, the study extracts video, audio, and text features and projects each modality into a common embedding space.The visual features combine pretrained 2D and 3D representations; all backbones remain fixed during training.Source: §4.1 · p. 5
4.2. Datasets
- Evaluation uses HowTo100M for training and YouCook2, MSR-VTT, CrossTask, and MiningYoutube for downstream assessment.The downstream datasets include human-annotated captions, action steps, or frame-level action labels.Source: §4.2 · pp. 5–6
4.3. Downstream Tasks
- Zero-shot evaluation covers text-to-video retrieval and temporal localization, comparing text with video and audio features and aligning frames to action labels.Retrieval uses averaged text-to-video and text-to-audio similarities; localization compares frame-level video-audio features with label embeddings.Source: §4.3 · p. 6
4.4. Comparison with State-of-the-art Methods
- MCN outperforms prior approaches on YouCook2 and MSR-VTT retrieval and on CrossTask and MiningYoutube localization, with especially strong CrossTask results on IOU and IOD.The comparisons include self-supervised baselines and, for CrossTask, a fully supervised approach.Source: §4.4 · pp. 6–7
4.5. Full Video Retrieval
- Caption averaging outperforms both majority-vote schemes for YouCook2 full-video retrieval, and MCN improves R@1 by 6.8% over prior work.On CrossTask full-video classification, it achieves 68.7% top-1 accuracy.Source: §4.5 · p. 7
4.6. Ablation Studies
- Ablations find clustering important for retrieval as well as clustering tasks, and report better results with the MMS contrastive loss.The clustering comparisons cover clustering methods, prediction targets, and hard versus soft labels; the authors report that the method brings modality features closer to semantic centroids.Source: §4.6 · pp. 7–8
4.7. Qualitative Analysis
- Qualitative results show correct open-vocabulary text-to-video retrieval and tighter, more separable action groupings with multimodal than visual-only features.The feature comparison uses CrossTask t-SNE visualizations; the retrieval examples are from YouCook2.Source: §4.7 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8
5. Conclusions
The conclusion presents a self-supervised multimodal clustering network that learns a common embedding space from local and global semantic relationships in narrated videos. It states that the approach can extend to additional modalities and other multimodal datasets.
5. Conclusions
- The network learns a common embedding space by processing local contrastive and global clustering semantic relationships in multimodal data.It is trained on narrated videos without manual annotations.Source: §5 · p. 8
- The authors identify optical flow and sentiment features as possible additional modalities and other multimodal datasets as potential applications.These are described as extensions of the approach, not evaluated results.Source: §5 · p. 8
Appendix
- The appendix is organized into method comparisons, experimental setup details, additional results, and quantitative results for two tasks.The tasks are text-to-video retrieval and temporal action localization.Source: Appendix · p. 9
A. Method description and comparisons.
MCN learns a joint representation across modalities, while XDC learns representations for individual modalities. The comparison highlights MCN’s use of contrastive and clustering losses and its rationale for including audio alongside video and text.
A.1. Salient features of the MCN method
- MCN combines contrastive and clustering losses, with the contrastive loss described as crucial for pulling together co-occurring instances across modalities.XDC uses only clustering loss.Source: §A.1 · p. 9
- MCN learns a joint representation across modalities, whereas XDC aims to learn representations for each modality for single-modality downstream tasks.MCN starts from pretrained feature extractors and learns projection heads across domains; XDC aims to learn feature backbones from scratch.Source: §A.1 · p. 9
- The authors describe MCN’s audio modality as a bridge between fine-grained video and text, which they characterize as a more abstract concept distilled from audio using ASR.They state that learning from all three modalities is beneficial.Source: §A.1 · p. 9
- The authors state that XDC’s visual and audio prediction targets differ, while its CDC method uses the same targets but is not evaluated on multimodal tasks.They therefore distinguish those approaches from MCN’s goal of learning a joint space across modalities.Source: §A.1 · p. 9
B.1. Implementation details
The implementation uses Adam with a cosine learning-rate schedule and runs for 30 epochs on four V100 GPUs. The reported setup also specifies a margin, batch size, and cluster size.
B.1. Implementation details
- Training runs for 30 epochs on four V100 GPUs and takes about two days.The optimizer is Adam with a 1e−4 learning rate and cosine learning-rate schedule.Source: §B.1 · p. 9
- The batch size is 4096 video clips and the cluster size is 256.Source: §B.1 · p. 9
- The margin hyperparameter δ is set to 0.001.Source: §B.1 · p. 9
B.2. Clustering metrics
The appendix evaluates clustering on CrossTask and MiningYouTube using standard metrics and reports that MiningYouTube follows a pattern similar to the CrossTask experiment. For temporal action localization, the number of clusters corresponds to the number of possible steps assigned to each test video.
B.2. Clustering metrics
- For temporal action localization, the number of k-means clusters k corresponds to the possible steps assigned to each video at test time.Source: §B.2 · p. 9
- The evaluation reports normalized mutual information (NMI), adjusted rand index (ARI), accuracy (Acc), mean entropy per cluster, and mean maximal purity per cluster.Estimated k-means pseudo-labels are matched to ground-truth targets using the Kuhn–Munkres/Hungarian algorithm.Source: §B.2 · p. 9
- Mean entropy is zero and mean purity is 100% under ideal conditions.Source: §B.2 · p. 9
- MiningYouTube clustering results show a pattern similar to the main-paper CrossTask experiment.The passage does not provide the table’s cell values.Source: §B.2 · p. 9
B.3. Clustering ablation explanation
The ablation examines clustering pipelines, pseudo-label sources, prediction targets, and label types; its alignment analysis suggests clustering can help separate aligned from misaligned multimodal instances.
B.3. Clustering ablation explanation
- Four pipelines vary Sinkhorn versus K-means clustering and swap versus joint prediction, with joint prediction using the mean feature.Figure 1 also contrasts the corresponding clustered-feature/pseudo-label and mean-pseudo-label/projected-feature losses.Source: §B.3 · p. 9, Figure 1 · p. 10
- K-means creates pseudo-labels as in deep clustering [11], while SeLa-style Sinkhorn clustering uses optimal transport [16] to encourage uniform cluster assignments.The uniform distribution is intended to prevent the learnable clustering network from degenerating.Source: §B.3 · p. 10
- Swap prediction uses another modality’s pseudo-label as target; joint prediction uses the mean feature’s pseudo-label to guide each modality’s features.K-means produces hard labels, while Sinkhorn outputs can be used as soft probabilities or converted to hard labels; soft-label prediction resembles SwAV [12] across modalities.Source: §B.3 · pp. 10–11
C.1. Dealing with miss-alignment across modalities
- With clustering, aligned instances are closer to the mean embedding and misaligned instances farther away, suggesting clustering may compensate for multimodal contrastive loss.The analysis compares cosine similarity to the mean multimodal embedding; pairwise MMS loss is lowest for audio-text and largest for video-text.Source: §B.3 · p. 11, §C.1 · p. 11
C.2. Ablation of modalities.
The modality ablations identify audio as a bridge between video and text, with results varying by dataset; the action-recognition evaluation also reports strong top-5 performance after action-related training.
C.2. Ablation of modalities.
- Audio is crucial for bridging video and text in the joint space, while the improvement on MSR-VTT is less pronounced than on YouCook2.The paper attributes this difference to dataset domain gaps: YouCook2 captions describe cooking steps, whereas MSR-VTT captions generally describe scenes and are less related to instructional ASR and audio.Source: §C.2 · p. 11
C.3. Zero-Shot Action Recognition
- MCN’s joint video-text space is sufficient for zero-shot action recognition on UCF-101 and HMDB despite training on instructional videos.The evaluation follows protocol [10] and reports top-1 and top-5 accuracy.Source: §C.3 · p. 12
- After training on a subset of HowTo100M, MCN achieves state-of-the-art Top-5 accuracy on both datasets; their names are not fully available in the cited passage.Baseline [10], designed specifically for zero-shot action recognition and trained using labeled action videos from Kinetics-700, leads to strong top-1 accuracy.Source: §C.3 · p. 12
C.4. CrossTask specific results.
CrossTask results vary by task and audio information; the accompanying analyses report the effects of fine-tuning, cluster count, and inference audio duration.
C.4. CrossTask specific results.
- CrossTask performance often improves over visual-only features, with the pattern varying across tasks in relation to the audio information available.The comparison is with visual-only features from [38] and.Source: §C.4 · p. 12
C.7. Audio length used in inference.
- 8 seconds of audio—4 seconds before and after—gives the best CrossTask inference results; longer segments can distract when steps are very short.The paper notes that some steps last less than 3 seconds.Source: Figure 2 · p. 12, §C.7 · p. 12
C.5. Finetune results
- Fine-tuning with YouCook training data yields significant improvement over the current baseline, although it requires ground-truth labels and is not the main focus.The fine-tune setting adds a training set provided by YouCook [52].Source: §C.5 · p. 12
C.6. Different number of clusters
- K-means cluster-size variations produce similar performance across datasets and evaluation metrics.The results are reported in Table 6.Source: §C.6 · p. 12
D. Qualitative analysis
The qualitative analysis highlights background separation in temporal localization and semantically similar results in text-to-video retrieval.
D. Qualitative analysis
- MCN distinguishes action steps from background scenes, with background frames often appearing as outliers relative to action-step clusters.The authors attribute the observed improvement to the clustering component.Source: D. Qualitative analysis · p. 12, Figure 3 · p. 14
- The MiningYouTube example aligns temporal boundaries to an action-step sequence given the video and sequence.Figure 3 illustrates the first minute of “Vegan Blueberry Quinoa Pancakes.”Source: Figure 3 · p. 14
- Retrieved text-to-video clips show similar semantics.Figure 4 presents retrieval examples.Source: D. Qualitative analysis · p. 12, Figure 4 · p. 14