PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Representations from Audio-Visual Spatial Alignment

Pedro Morgado, Yi Li, Nuno Vasconcelos

arXiv:2011.01819v1cs.CV

TL;DR

Video-level audio-visual matching and synchronization do not use spatial cues that can distinguish co-occurring visual concepts. The paper trains representations by contrastively aligning spatial audio and 360° video across viewpoints, and reports benefits across several downstream tasks.

  • Problem

    Video-level audio-visual correspondence and temporal synchronization disregard spatial cues and can leave co-occurring visual concepts, such as cars and roads, difficult to distinguish.

    Source: §1 · pp. 1–2

  • Method

    AVSA contrastively trains on 360° video and spatial audio, distinguishing clips from different viewpoints and combining viewpoint representations with a transformer.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    AVSA yields better representations than prior audio-visual self-supervision across downstream tasks and exceeds AVC and AVTS by more than 5% on spatial alignment.

    Source: §5.2 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    The authors report benefits for audio-visual correspondence and spatial alignment, action recognition, and video semantic segmentation.

    Source: §1 · p. 2, §6 · p. 8

    AVSA requires 360° data with spatial audio, which is less prevalent than regular video.

    Source: §6 · p. 9

Abstract

from arXiv · show

We introduce a novel self-supervised pretext task for learning representations from audio-visual content. Prior work on audio-visual representation learning leverages correspondences at the video level. Approaches based on audio-visual correspondence (AVC) predict whether audio and video clips originate from the same or different video instances. Audio-visual temporal synchronization (AVTS) further discriminates negative pairs originated from the same video instance but at different moments in time. While these approaches learn high-quality representations for downstream tasks such as action recognition, their training objectives disregard spatial cues naturally occurring in audio and visual signals. To learn from these spatial cues, we tasked a network to perform contrastive audio-visual spatial alignment of 360{\deg} video and spatial audio. The ability to perform spatial alignment is enhanced by reasoning over the full spatial content of the 360{\deg} video using a transformer architecture to combine representations from multiple viewpoints. The advantages of the proposed pretext task are demonstrated on a variety of audio and visual downstream tasks, including audio-visual correspondence, spatial alignment, action recognition, and video semantic segmentation.

1 Introduction

Prior audio-visual self-supervision uses video-level matching or temporal synchronization, but does not use spatial cues to resolve which co-occurring visual concepts produce a sound. This work learns representations by spatially aligning audio and video from different viewpoints in 360° content.

1 Introduction

  • AVC matches audio and video from the same or different videos, while AVTS uses out-of-sync clips from the same video as hard negatives.These approaches learn from video-level correspondence or timing rather than spatial cues.Source: §1 · p. 1, Figure 1 · p. 2
  • Video-level audio labels can confuse co-occurring concepts such as cars and roads, limiting representations that distinguish the sound-producing object.Prior work also demonstrated poor audio localization with AVC pretraining.Source: §1 · pp. 1–2
  • The paper introduces AVSA, which trains on 360° video and spatial audio by treating audio and video clips from different viewpoints as negatives.The collected 360° dataset is five times larger than currently available datasets; training also uses a curriculum and a transformer to combine multiple viewpoints.Source: §1 · p. 2
  • The authors report benefits of AVSA for audio-visual correspondence and spatial alignment, action recognition, and video semantic segmentation.Source: §1 · p. 2

2 Related work

Related work situates the approach among 360° vision, self-supervised and contrastive learning, spatial-context objectives, and audio-visual representation learning.

2 Related work

  • 360° vision research includes viewpoint recognition, depth and camera-motion estimation, salient-view guidance, field-of-view control, and spatializing mono audio.Source: §2 · p. 2
  • Self-supervised learning learns representations without explicit human annotation; contrastive learning uses losses to learn view-invariant representations from augmented, temporal or spatial, and co-occurring-modality views.Source: §2 · p. 2
  • Prior spatial-context objectives predict relative patch locations or spatiotemporal position, but visual-only inputs and targets can permit low-level shortcuts instead of semantic features.The paper positions co-occurring spatial audio as a way to supply spatial context from another modality.Source: §2 · pp. 2–3
  • Audio-visual correspondence has been used for object, scene, action, synchronization, and audio-classification tasks, but matching audio to whole clips offers little incentive to distinguish often-co-occurring objects.The paper explores spatial cues in 360° video and spatial audio to encourage reasoning about scene composition and sound-source locations.Source: §2 · p. 3

3 Audio-visual spatial alignment

AVSA learns from spatial cues in 360° video and spatial audio by aligning viewpoint-specific crops. Its architecture combines information across viewpoints, and training moves from video-level audio-visual correspondence to harder crop-level matching.

3.1 Pretext task

  • Regression of the applied rotation is rejected because its continuous target is difficult to optimize and defining the task on full 360° video limits aggressive cropping.Aggressive cropping is identified as critical for self-supervised learning.Source: §3.1 · p. 3
  • K viewpoint-centered video crops are paired with audio realigned to each crop’s direction, and the network learns their correct audio-visual correspondence.The video crops use a Gnomonic projection; audio is reoriented so its frontal direction points toward the crop’s viewing angle.Source: §3.1 · p. 3

3.2 Architecture

  • Because audio includes sources from all directions while each video crop shows only nearby sources, cross-modal translation must combine features from all sampled viewpoints.The translation networks use a transformer that alternates attention over the K features with clip-wise feed-forward transformations.Source: §3.2 · p. 4
  • The translation transformer uses one attention head and omits positional encodings, since encoding viewing directions could let the model solve alignment without learning semantic representations.Early empirical results found no downstream-task improvement from multi-head attention.Source: §3.2 · p. 4

3.3 Learning strategy

  • Training begins with video-level AVC from one randomly angled crop per video, then advances to crop-level matching with multiple views from the same video.The second phase adds audio-to-video and video-to-audio translation networks and trains translated features to match encoder outputs for the corresponding viewing angle.Source: §3.3 · pp. 4–5
  • AVC uses the opposing-modality feature from the same video as its target and uses target representations from videos in the batch as its proposal distribution.The prediction head computes cosine similarity after linear projection into a low-dimensional space, with τ as the temperature hyper-parameter.Source: §3.3 · p. 5

4 YouTube-360 dataset

The YouTube-360 dataset contains diverse 360° videos with spatial audio, collected and filtered for representation learning. It also provides pseudo-labels for evaluating semantic segmentation.

4 YouTube-360 dataset

  • YT-360 contains 5,506 videos—4,506 for training and 1,000 for testing—and 88,733 clips of roughly 10 seconds, totaling 246 hours.The clips were selected above a volume threshold to exclude periods of silence.Source: §4 · p. 5
  • The dataset spans topics including musical performances, vlogs, and sports, and is five times larger than the largest previously collected 360° video dataset.Videos lacking valid ambisonics, containing only still images, or dominated by post-production sound were removed.Source: §4 · p. 5
  • Because ground-truth annotations were impractical to collect, a ResNet101 Panoptic FPN model trained on MS-COCO generated segmentation maps for 32 frequent objects and background classes.These maps were used to evaluate AVSA representations through knowledge distillation.Source: §4 · p. 5

5 Experiments

Experiments evaluate AVSA representations on binary audio-visual tasks, semantic segmentation, and action recognition. Across these evaluations, the reported results favor AVSA over AVC and AVTS, with further gains from multi-view context in several settings.

5.1 Experimental setting

  • Experiments used four NFOV crops per video, first-order ambisonics, an 18-layer R2+1D video encoder, and a 9-layer audio encoder.The AVC and AVTS baselines were trained on YouTube-360 with the same architecture and optimization settings; AVTS included negatives from different times.Source: §5.1 · p. 6

5.2 Audio-visual spatial alignment

  • AVSA outperformed AVC and AVTS on both binary evaluations, by more than 6% over AVC on AVC-Bin and more than 5% over both on AVSA-Bin.Four-view predictions improved AVSA by 5.5% on AVC-Bin and 4% on AVSA-Bin; the transformer improved results in 5 of 6 configurations.Source: §5.2 · p. 7, Table 2 · p. 7

5.3 Semantic segmentation by knowledge distillation

  • With four-view context features, AVSA achieved 3% higher mIoU than AVC and 1% higher than AVTS on semantic segmentation.AVSA also improved over AVC particularly on background classes, including rocks (34.7% vs. 27.7% accuracy) and sea (50.1% vs. 46.8%).Source: §5.3 · p. 7, Table 3 · p. 8
  • Without curriculum learning or transformer-based cross-view context, AVSA each yielded 1.5% worse mIoU than the full model.Replacing the transformer with a similarly sized multilayer perceptron affected context-feature performance more than visual representations alone.Source: §5.3 · pp. 7–8, Table 3 · p. 8

5.4 Action recognition

  • AVSA dense predictions outperformed AVC by 4% on UCF and 3% on HMDB, and AVTS by 3.5% on UCF and 2% on HMDB.The evaluation reports top-1 accuracy for single-clip predictions and averages over 25 clips per video.Source: §5.4 · p. 8, Table 4 · p. 8

6 Discussion, future work and limitations

AVSA uses spatial cues to learn representations that perform better on downstream tasks, while viewpoint reasoning with a transformer benefits tasks defined on 360° video. Its discussion also notes a data-availability constraint and the potential value of more targeted clip sampling.

6 Discussion, future work and limitations

  • AVSA yields better representations than prior audio-visual self-supervision for audio-visual correspondence, video semantic segmentation, and action recognition.Source: §6 · p. 8
  • Summarizing information across viewpoints with a transformer benefits downstream tasks defined on 360° video.Source: §6 · pp. 8–9
  • GradCAM visualizations qualitatively show AVSA models localizing sound sources better than AVC models.Figure 7 compares sound-localization maps from AVC and AVSA models.Source: §6 · p. 9, Figure 7 · p. 9
  • More sophisticated sampling is future work; objectness, optical flow, or audio energy could guide selection away from background-dominated views that may harm representation quality.Source: §6 · p. 9
  • AVSA requires 360° video with spatial audio, which is less prevalent than regular video; prior methods trained on much larger datasets can perform better on action recognition.For the same amount of training data, AVSA improves learned representation quality significantly.Source: §6 · p. 9

Broader Impact

The authors note that self-supervision reduces reliance on human labels but does not prevent learned representations from reflecting collection biases.

Broader Impact

  • Self-supervision reduces the need for human labeling, which is in some sense less affected by human biases.Source: Broader Impact · p. 9
  • Self-supervised models still reflect biases present in the data collection process.Source: Broader Impact · p. 9
  • The authors searched for 360° videos using queries translated into multiple languages, but unequal adoption of 360° cameras across societal sectors may leave representation discrepancies.Source: Broader Impact · p. 9

A Implementation details

This section describes implementation of AVSA pre-training and the semantic segmentation and action recognition downstream tasks.

A Implementation details

  • The implementation section covers AVSA pre-training alongside semantic segmentation and action recognition downstream tasks.Source: A Implementation details · p. 12

A.1 Audio-visual spatial alignment

The implementation specifies encoder and feature-translation architectures, training objectives and negative-pair construction, and points to the pre-training hyperparameter table.

A.1 Audio-visual spatial alignment

  • The video and audio encoder architectures are shown in Table 5, while the feature translation networks are described in Section 3.2 and Figure 3.The feature-translation transformers have base dimension 512 and expansion ratio 4; W1 outputs 2048 dimensions while Wkey, Wqr, Wval, W0, and W2 output 512.Source: §A.1 · p. 12
  • AVC and AVTS/AVSA models optimize different losses; AVTS negatives use the same viewpoint at different times, whereas AVSA negatives use different viewpoints at the same time.Source: §A.1 · p. 12
  • Models use Adam, and pre-training hyperparameters are summarized in Table 6.Source: §A.1 · p. 12

A.2 Semantic segmentation

The section describes the semantic-segmentation head and the action-recognition evaluation setup.

A.2 Semantic segmentation

  • Semantic segmentation uses a lightweight FPN head with temporal pooling, multiscale feature fusion, and pixel-wise classification.Pixels with ground-truth confidence below 75% are excluded from training labels and evaluation; the head is trained for 10 epochs.Source: §A.2 · p. 13

A.3 Action recognition

  • Action recognition evaluates the video encoder on UCF and HMDB using a finetuned linear-classification model.Performance is reported on the first train/test split for each dataset.Source: §A.3 · p. 13

B Ablations and parametric studies

The ablations test spatial audio, viewpoint count, negative-crop strategy, curriculum learning, and transformer-based spatial context across pretext and downstream tasks.

B Ablations and parametric studies

  • The ablation evaluation covers AVC-Bin, AVSA-Bin, semantic segmentation, and action recognition, using accuracy, mean IoU, and clip-level accuracy measures.The audio network accepts mono, stereo, or ambisonic spectrogram inputs; AVSA models are initialized from an AVC model at epoch 100.Source: B Ablations and parametric studies · p. 13, Table 5 · p. 15, Table 6 · p. 15

Influence of spatial audio

  • Approximately 7% improvement in AVC and AVSA tasks comes from full ambisonics over mono audio.Stereo also improves over mono, but by a smaller margin; ambisonics is more effective for semantic segmentation and UCF action recognition as well.Source: B Ablations and parametric studies · p. 13

Influence of number of viewpoints

  • AVC and AVSA performance rises monotonically with viewpoint count, while semantic-task optima differ by task.Segmentation performs best with 2 or 4 viewpoints, and action recognition peaks at 4 viewpoints.Source: B Ablations and parametric studies · pp. 13–14

Influence of type of negative crops

  • Instance-based and spatial negatives together outperform instance-based negatives alone, while adding temporal negatives yields mixed downstream results.Temporal negatives slightly improve semantic segmentation but worsen UCF performance.Source: B Ablations and parametric studies · p. 14

Influence of curriculum learning

  • Hard-only negatives yield the best AVC and AVSA performance, but the representations transfer poorly to semantic tasks.Combining easy and hard negatives benefits segmentation and action recognition, with curriculum learning achieving the best results on those tasks.Source: B Ablations and parametric studies · p. 14

Influence of modeling spatial context

  • Removing the transformer makes spatial alignment harder and also affects semantic segmentation and action recognition.A transformer of depth D = 2 provides a good trade-off between model complexity and performance for semantic tasks.Source: B Ablations and parametric studies · p. 14

Related papers

Loading 2011.01819v1…