Source-linked AI summary
Labelling unlabelled videos from scratch with multi-modal self-supervision
Yuki M. Asano, Mandela Patrick, Christian Rupprecht, Andrea Vedaldi
TL;DR
The paper addresses the high cost of manually labelling videos and the limited evidence for unsupervised video clustering. It proposes SeLaVi, which jointly learns clusters and representations using audio-visual correspondence, and reports semantically strong clusters across common video datasets.
Problem
Video annotation is costly, while unsupervised image labelling has advanced further than unsupervised video labelling.
Method
SeLaVi jointly learns representations and clusters, adapting SeLa with nonuniform cluster priors and audio-visual modalities treated as augmentations.
Results
SeLaVi substantially outperforms alternative clustering methods across four video datasets, including 55% perfectly grouped videos on AVE and 32% on VGG-Sound.
Takeaways & Limitations
The results establish benchmark baselines and show that multimodal clustering can produce highly semantic video labels without human annotations.
Takeaways & Limitations
Users may overestimate clustering quality because the method relies on basic statistical properties and convolutional-network inductive biases.
Abstract
from arXiv · showhide
A large part of the current success of deep learning lies in the effectiveness of data -- more precisely: labelled data. Yet, labelling a dataset with human annotation continues to carry high costs, especially for videos. While in the image domain, recent methods have allowed to generate meaningful (pseudo-) labels for unlabelled datasets without supervision, this development is missing for the video domain where learning feature representations is the current focus. In this work, we a) show that unsupervised labelling of a video dataset does not come for free from strong feature encoders and b) propose a novel clustering method that allows pseudo-labelling of a video dataset without any human annotations, by leveraging the natural correspondence between the audio and visual modalities. An extensive analysis shows that the resulting clusters have high semantic overlap to ground truth human labels. We further introduce the first benchmarking results on unsupervised labelling of common video datasets Kinetics, Kinetics-Sound, VGG-Sound and AVE.
1 Introduction
The paper targets unsupervised video clustering, where annotation is costly and strong representations alone do not guarantee useful clusters. It introduces SeLaVi, which jointly learns representations and clusters while exploiting audio-visual correspondence.
- Video clustering is especially challenging because videos are expensive to annotate and combine temporal information with audio and visual modalities.
- Post-hoc clustering of self-supervised video representations provides strong baselines without manually supplied labels.
- SeLaVi jointly learns video representations and clusters, showing that good clusters do not come automatically from good representations.
- The method extends SeLa by allowing nonuniform cluster probabilities, treating audio and visual streams as augmentations, and synchronizing their initialization.
- SeLaVi substantially outperforms post-hoc clustering, SeLa applied to frames, and XDC across four video datasets.
- 32% of VGG-Sound videos and 55% of AVE videos are perfectly grouped without labels during training.
- The work benchmarks unsupervised video labelling, develops clustering baselines, and introduces a multimodal algorithm producing highly semantic labels.
2 Related work
Prior work spans unsupervised image clustering, video representation learning, and post-hoc video clustering. Video methods increasingly exploit temporal structure and multiple modalities, but clustering remains a distinct challenge.
- Image-clustering methods combine feature learning with clustering through transformations, explicit cluster objectives, or strong pretrained representations.
- Video representation learning uses temporal tasks and multimodal signals including audio, speech, optical flow, synchronized timing, and ASR-generated text.
- Figure 1 frames the paper’s approach as multimodal clustering from scratch, treating modalities as distinct augmentations that can produce clusters aligned with human labels.
- Post-hoc video clustering applies a clustering algorithm to features from a pretrained representation and can yield small but consistent gains when clusters are reused as labels.
3 Method
The method learns video labels without ground-truth annotations by jointly learning representations and clusters, while addressing degenerate assignments, skewed class frequencies, and multimodal correspondence. It extends optimal-transport clustering to audio-visual data and adds mechanisms for arbitrary priors, modality-invariant labels, initialization alignment, and decorrelated clustering heads.
- Problem formulation: The labelling function assigns semantic cluster labels to multimodal data without ground-truth annotations, aiming to reproduce labels that human annotators would associate with videos.The method is illustrated for videos containing audio and visual streams, though it can support other modalities.
- Non-degenerate clustering via optimal transport: Jointly optimizing clustering and representation learning can collapse to a constant representation, producing unbalanced or degenerate solutions.The method therefore constrains cluster assignments rather than optimizing an unconstrained clustering energy directly.
- Non-degenerate clustering via optimal transport: SeLa converts clustering into an optimal-transport problem by constraining cluster marginals and alternating Sinkhorn-Knopp assignment updates with neural-network training.The transportation formulation uses a matrix Q whose marginals correspond to clusters and data indices.
- Clustering with arbitrary prior distributions: The method replaces uniform cluster probabilities with an arbitrary prior distribution and identifies the required cluster permutation by sorting, at O(K log K) cost.This accommodates highly unbalanced distributions that may better match real-world video classes.
- Multi-modal single labelling: Multimodal single labelling learns separate audio and visual representations that produce the same clusters, making assignments invariant to modality choice and standard augmentations.The two output layers are synchronized before training by permuting one network’s final layer, and decorrelated clustering heads address multiple valid semantic organizations.
4 Experiments
The experiments evaluate unsupervised video labelling across datasets, modalities, ablations, and downstream retrieval. SeLaVi is compared with representation-based clustering baselines and achieves strong semantic alignment on three datasets, while also learning useful visual representations.
- Experimental design: Experiments cover modality analysis, ablations, unsupervised labelling, and downstream audio-visual representation learning.The datasets and evaluation protocols include VGG-Sound, Kinetics-400, Kinetics-Sound, and AVE, with NMI, ARI, accuracy, entropy, and cluster purity metrics.
- Experimental design: The baselines combine pretrained video representations with k-means, including supervised and self-supervised encoders.The comparison includes DPC, MIL-NCE, XDC, and frame-based or representation-based clustering approaches.
- Multi-modality: 57% NMI is achieved by SeLaVi using only visual input, surpassing MIL-NCE at 49% NMI; combining audio and visual modalities performs significantly better than either alone.Audio-only performance drops only slightly, indicating that both modalities contribute to the method’s clustering performance.
- Multi-modality: SeLaVi degrades more gracefully than baselines when video resolution is compressed and decompressed by factors of 1, 4, 8, and 16.The analysis interprets this robustness as evidence that the method relies on both modalities.
- Unsupervised labelling: 55.9%, 41.2%, and 57.9% accuracy are obtained on VGG-Sound, Kinetics-Sound, and AVE, respectively, while Kinetics-400 clusters are not well aligned with human labels.The Kinetics-400 mismatch is attributed to annotations focused strongly on visual human actions, whereas the method can group clips by audible context.
- Downstream representations: Unsupervised visual representations improve retrieval performance without finetuning, exceeding the prior state of the art by over 100% for Recall @1 on UCF-101 and HMDB-51.The downstream evaluation is presented as a side effect rather than the paper’s primary goal.
5 Conclusion
The work establishes strong unsupervised video-labelling baselines and introduces a multimodal approach that outperforms competing methods on the evaluated benchmarks.
- The paper establishes strong baselines for unsupervised labelling across several popular video datasets.
- Its simultaneous clustering and representation-learning approach for multimodal data outperforms other methods on these benchmarks.
- The analysis examines the importance of multimodality for unsupervised video labelling.
- Strong representations alone are not sufficient for obtaining good clustering results.
- The strongest feature representations remain those obtained through supervised, well-clustered training.
Broader Impact
The method is presented as a way to reduce human annotation effort by grouping videos through audio-visual evidence, while its quality may be overestimated in expert-dependent domains.
- Unsupervised clustering can reduce annotation effort when consistent clusters allow one manual label to be propagated to other videos.
- The method groups videos using evidence from both audio and visual streams to produce consistent, homogeneous groupings.
- Users may overestimate clustering quality because the method cannot incorporate the deep domain expertise of human specialists.
- Its clusters may differ from expert-defined groupings in particular domains and may exploit undesirable dataset biases.
A.1 Pretrained model details
This appendix section points to additional information about the pretrained models and identifies the audio encoder details used for XDC benchmarking.
- The appendix provides additional information about the pretrained models used in the work.
- Table A.1 covers the architectural and pretraining details of XDC’s audio encoder used for benchmarking.
A.2 Implementation details
The implementation uses specified clustering, optimization, distributional, and input settings for training and evaluation.
- Training uses Sinkhorn-Knopp parameter λ = 20, an inverse quadratic schedule with 100 clustering operations, and 10 clustering heads.
- Evaluation reports head 0 against ground truth because no significant performance difference was found between heads.
- The Gaussian distribution uses marginals from N(1, 0.1) ∗N/K, while clustering heads use two-layer MLPs.
- The video inputs contain 30 frames.
A.3 Pair-based optimization for AV-Alignment
The visual and audio encoders are aligned by searching over feasible pair switches and selecting the lowest-cost permutation.
- The method evaluates 50,000 potential pair switches across five randomized restarts and retains the permutation with the lowest cost.
A.4 Evaluation metrics details
The section defines clustering metrics for comparing unsupervised clusters with ground-truth labels and reports robustness to the estimated number of clusters. It also describes downstream retrieval and fine-tuning evaluations on standard video benchmarks.
- Clustering metrics: NMI measures mutual information between predicted and ground-truth clusterings, ranging from 0 for no mutual information to 100% for perfect correlation.
- Clustering metrics: ARI corrects the Rand index for random assignments, with values near 0 indicating random correlation and 1 indicating identical agreement.
- Clustering metrics: Mean cluster entropy is optimal at 0, while chance assignment yields ⟨H⟩= −log 1/K.
- Clustering metrics: Mean maximal purity ranges from chance level, ⟨pmax⟩= 1/K, to perfect matching at ⟨pmax⟩= 100%.
- Robustness and evaluation protocols: 38.0 to 42.7: increasing K from 309 to 619 raises mean maximal purity, while NMI remains almost constant and ARI declines for larger K.The authors interpret the results as showing robustness to different estimates of the ground-truth class count.
- Robustness and evaluation protocols: Downstream evaluation uses nearest-neighbor retrieval and whole-network fine-tuning on UCF-101 and HMDB-51 under standard protocols.Retrieval reports R@k, while representation learning reports average top-1 accuracy over official splits.