PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Learning by Cross-Modal Audio-Video Clustering

Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, Du Tran

arXiv:1911.12667v3cs.CV

TL;DR

The paper addresses whether correlated but distinct audio and video signals can improve self-supervised representation learning beyond within-modality methods. It proposes XDC, which uses one modality’s clusters to supervise the other, and reports that XDC outperforms competing approaches, including fully supervised pretraining for action recognition. The learned representations may nevertheless inherit biases from the training data.

  • Problem

    Manually labeled video datasets are expensive to scale, and differing action label spaces limit their transferability across datasets.

  • Method

    XDC alternates clustering one modality’s deep features with using those cluster assignments as pseudo-labels to train the encoder for the other modality.

  • Results

    XDC outperforms existing self-supervised methods and fully supervised ImageNet- and Kinetics-pretraining for action recognition.

  • Takeaways & Limitations

    Cross-modal supervision produces representations that generalize across downstream action-recognition and audio-classification benchmarks.

  • Takeaways & Limitations

    Because XDC is self-supervised, its learned model may exhibit biases intrinsically present in the training data.

Abstract

from arXiv · show

Visual and audio modalities are highly correlated, yet they contain different information. Their strong correlation makes it possible to predict the semantics of one from the other with good accuracy. Their intrinsic differences make cross-modal prediction a potentially more rewarding pretext task for self-supervised learning of video and audio representations compared to within-modality learning. Based on this intuition, we propose Cross-Modal Deep Clustering (XDC), a novel self-supervised method that leverages unsupervised clustering in one modality (e.g., audio) as a supervisory signal for the other modality (e.g., video). This cross-modal supervision helps XDC utilize the semantic correlation and the differences between the two modalities. Our experiments show that XDC outperforms single-modality clustering and other multi-modal variants. XDC achieves state-of-the-art accuracy among self-supervised methods on multiple video and audio benchmarks. Most importantly, our video model pretrained on large-scale unlabeled data significantly outperforms the same model pretrained with full-supervision on ImageNet and Kinetics for action recognition on HMDB51 and UCF101. To the best of our knowledge, XDC is the first self-supervised learning method that outperforms large-scale fully-supervised pretraining for action recognition on the same architecture.

1 Introduction

The paper motivates self-supervised video learning through audio-video relationships, addressing the cost and ambiguity of manually labeled action-recognition datasets. It proposes cross-modal approaches that use information between modalities to learn video representations.

  • Motivation: Perceptual studies show that sounds provide important cues for interpreting actions and dynamic visual events.Examples include altered perception of object motion and sound-guided spatiotemporal visual attention.
  • Approach: The work investigates whether spatiotemporal action-recognition models can be pretrained from unlabeled videos by capturing cross-modal audio-video information.
  • Motivation: Manual annotation is costly to scale, while action-recognition datasets use substantially different label spaces.These differences create a label-space gap between pretrained source datasets and smaller target datasets.
  • Results: All three cross-modal methods generalize better than within-modality counterparts on downstream action-recognition and audio-classification tasks.
  • Results: XDC outperforms the other approaches and uses the other modality as an exclusive supervisory signal.
  • Results: XDC pretrained on large-scale unlabeled video outperforms fully supervised ImageNet and Kinetics pretraining for HMDB51 and UCF101 action recognition.The authors identify this as the first such result for self-supervised video representation learning on the same architecture.

2 Related work

Prior self-supervised work includes reconstruction, image-based pretext tasks, and cross-modal objectives. The paper extends deep clustering to multimodal video by introducing a framework summarized through its model variants.

  • Unsupervised representation learning: Earlier unsupervised representation-learning methods commonly learn by reconstructing their inputs.The paper instead frames its approach as self-supervised learning through pseudo-labels generated for supervised training.
  • Self-supervised learning: Image and video self-supervised methods use pretext tasks such as spatial-context prediction and jigsaw solving.
  • Deep clustering: The framework compares single-modality deep clustering with Multi-Head, Concatenation, and Cross-Modal Deep Clustering.Figure 1 presents SDC, MDC, CDC, and XDC as alternative ways to cluster features and generate self-supervision signals.
  • Cross-modal learning: Earlier cross-modal methods include audio-visual correspondence, aligned representations, temporal synchronization, contrastive multiview coding, and ambient-sound learning.

3 Technical approach

The method adapts iterative deep clustering to visual and audio encoders. Its variants differ in whether they cluster each modality separately, concatenate modalities, or supervise each encoder exclusively with the other modality’s clusters.

  • Single-modality foundation: DeepCluster iteratively clusters deep features from one encoder and trains that encoder using the resulting cluster assignments.
  • Multimodal setup: The multimodal setting uses visual and audio encoders that produce separate visual and audio feature sets from unlabeled video clips.
  • Framework: The multimodal framework defines three approaches—MDC, CDC, and XDC—for adapting deep clustering to visual and audio inputs.The approaches are described as different steps within each deep-clustering iteration.
  • MDC: MDC gives each encoder two classification heads supervised by pseudo-labels from its own modality and the other modality.
  • CDC: CDC clusters concatenated, separately l2-normalized visual-audio feature vectors and uses those assignments to update both encoders.
  • XDC: XDC clusters audio features to supervise the visual encoder and visual features to supervise the audio encoder.Each encoder relies exclusively on clusters learned from the other modality.

4 Experiments

The experiments evaluate XDC across pretraining datasets, downstream benchmarks, clustering choices, data curation, and transfer strategies. XDC benefits from multimodal supervision, larger pretraining datasets, and cross-modal representations that remain useful under feature extraction and finetuning.

  • Experimental setup: 240K, 2M, and 65M videos are used in Kinetics, AudioSet, and Instagram pretraining datasets, while downstream evaluation covers UCF101, HMDB51, and ESC50.UCF101 and HMDB51 measure action recognition; ESC50 measures sound classification.
  • Experimental setup: Top-1 accuracy on split-1 is reported for self-supervised clustering models and Scratch, Supervised Pretraining, and same-modality-XDC baselines.The compared models use common visual and audio encoders but differ in their self-supervision.
  • Single-modality vs. multi-modal clustering: 12.4%, 7.6%, and 11.5% are the maximum multi-modal improvements over single-modality deep clustering on UCF101, HMDB51, and ESC50, respectively.XDC achieves the best performance across all three downstream tasks in this ablation.
  • Number of clusters: The best k-means cluster count tends to increase with pretraining data size, including a shift from 128 to 256 for HMDB51 when moving from AudioSet-240K to AudioSet.The authors hypothesize that larger datasets provide more diverse samples suitable for finer-grained clusters.
  • Pretraining data type and size: 9.8%, 22.2%, and 24.1% are HMDB51 improvements for XDC pretrained on AudioSet, IG-Random, and IG-Kinetics, respectively, compared with Kinetics pretraining.The study reports improvements across all three downstream tasks as pretraining data size increases.
  • Curated vs. uncurated data: 5.2%, 2.1%, and 1.9% are the HMDB51 curated-versus-uncurated performance gaps at 1M, 16M, and 65M videos, respectively.At larger scale, uncurated data remains meaningful, while curated data is positively biased toward human-action semantics and can hurt ESC50 performance.
  • Transfer strategy: XDC remains useful both as a fixed feature extractor and as a finetuning initialization, with fixed XDC features outperforming several fully-finetuned supervised models.Most pretrained models perform worse when frozen, but XDC’s relative performance against supervised models generally remains similar.

5 Understanding XDC

XDC learns semantically coherent audio and video clusters, while allowing the two modalities to organize related actions differently. Its learned label space need not match existing dataset taxonomies to produce strong general features.

  • Top-purity XDC clusters group semantically related actions, including scuba diving with snorkeling in audio and feeding fish in video.The examples are ranked by purity with respect to Kinetics labels and represented by videos nearest each cluster centroid.
  • Audio and video clusters capture different relationships among visually or acoustically related activities.Scuba diving is grouped with snorkeling by audio features but with feeding fish by visual features.
  • XDC clusters also organize motor-engine sounds, swimming strokes, golf shots, and cooking activities.These examples illustrate coherence beyond the scuba-diving clusters.
  • Even clusters with low Kinetics purity can retain scene-level coherence, such as farms in audio and gym activities in video.The paper notes that many other bottom-ranked clusters lack coherence when viewed through Kinetics labels.
  • XDC’s learned label space yields strong and general audio and video features despite imperfect alignment with existing dataset taxonomies.

6 State-of-the-art self-supervised learning comparison

XDC is evaluated against self-supervised and fully supervised pretraining on video action recognition and audio event classification benchmarks. It achieves leading results across these comparisons, with advantages that increase over AVTS at larger pretraining scale.

  • The reported comparisons use finetuning for UCF101 and HMDB51, while XDC models are publicly released.
  • XDC pretrained on IG-Kinetics sets new self-supervised state of the art on UCF101 and HMDB51.It outperforms Elo by 1.7% on UCF101 and 1.5% on HMDB51.
  • XDC on IG-Kinetics outperforms fully supervised Kinetics pretraining by 1.3% on UCF101 and 3.8% on HMDB51.
  • When both methods use AudioSet, XDC outperforms AVTS by 3.9% on UCF101 and 5.6% on HMDB51.The paper interprets these larger margins as evidence that XDC scales better than AVTS.
  • XDC achieves state-of-the-art performance on DCASE and competitive ESC50 results with a 1.1% gap to.

7 XDC for temporal action localization

The paper tests XDC features in temporal action localization by supplying them to G-TAD without finetuning the feature extractors. Both XDC variants outperform fully supervised features across all evaluated temporal overlap thresholds.

  • Both XDC feature variants outperform fully supervised Kinetics features across all tIoU thresholds on THUMOS14.Performance is reported using mean Average Precision (mAP) at different temporal Intersection over Union thresholds.
  • G-TAD uses frozen XDC features from R(2+1)D-18 models pretrained on IG-Kinetics or IG-Random.The feature extractors are not finetuned in this experiment.

8 Conclusion

The paper presents XDC as a self-supervised video-and-audio model that outperforms existing self-supervised methods and fully supervised ImageNet- and Kinetics-pretraining for action recognition.

  • XDC outperforms existing self-supervised methods for action recognition.
  • XDC also surpasses fully supervised ImageNet- and Kinetics-pretraining for action recognition.
  • The comparison uses the same architecture and a larger number of uncurated videos for self-supervised pretraining.

Broader Impact Statement

The broader impact statement describes video representation learning as broadly applicable while warning that self-supervised models may inherit biases from their training data.

  • Video semantic representations have potential applications in entertainment, communication, and teaching.
  • The authors do not foresee major ethical issues associated with this work.
  • Because the approach is self-supervised, its learned model may exhibit biases intrinsically present in the training data.

Supplementary Material

The supplementary material details optimization, initialization, clustering, and the modality choices underlying XDC.

  • The optimization section contrasts the single-modality baseline with multi-modal models and describes the full optimization cycle.
  • Trivial solutions: SDC may converge to trivial solutions, including empty clusters or classifiers that predict the same label for every input.
  • Initialization and convergence: Initial pseudo-labels are obtained by clustering features from randomly initialized encoders, which can capture weak input similarities.
  • Clustering and optimization schedule: XDC primarily uses k-means because of its efficiency and scalability, while the cluster count controls pretext-task difficulty.
  • The method can also adapt to modalities such as text, whereas audio supports environmental sounds and arbitrary uncurated Web videos.

C Hyperparameters and training details

The training details cover distributed optimization, parameter definitions, stopping schedules, and fair downstream finetuning, alongside Kinetics experiments with ResNet3D-18.

  • Training: Models are trained with distributed SGD on a GPU cluster using a warmup scheme.
  • Training parameter definitions: The supplementary material provides definitions and descriptions for the training parameters.
  • Pretraining parameters: Early stopping is used on Kinetics and AudioSet because of observed overfitting on the pretext tasks.
  • Pretraining parameters: For IG-Kinetics and IG-Random, the final XDC iteration is pretrained three times longer.
  • Finetuning parameters: Finetuning compares methods using cross-validated learning rates from the same candidate set.
  • XDC with a ResNet3D-18 visual backbone outperforms training from scratch on three downstream tasks.

E Additional qualitative results

Additional analyses examine XDC’s learned filters, clusters, backbone transfer, and finetuning evaluation choices. XDC produces more diverse motion filters than fully supervised pretraining, while the supplementary tables document cluster purity and comparison protocols.

  • XDC clusters: Tables 14 and 15 rank XDC audio and video clusters by purity relative to Kinetics labels and list each cluster’s five most frequent concepts.The tables report the top and bottom ten clusters for both modalities.
  • Finetuning: Finetuning comparisons cross-validate all pretraining methods over the same base-learning-rate range and report each method’s best result.The finetuning protocol fixes γ at 0.01; learning FC-only benefits from a wider base-learning-rate range.
  • Different backbone: XDC is evaluated with ResNet3D-18 against scratch and fully supervised Kinetics-pretrained baselines using split-1 top-1 accuracy.This analysis tests whether the method transfers beyond the primary visual backbone.
  • XDC filters: XDC learns a more diverse set of motion filters than fully supervised R(2+1)D pretraining.The comparison uses conv_1 spatial and temporal filters from IG-Kinetics self-supervised XDC and Kinetics fully supervised training.
Loading 1911.12667v3…