Source-linked AI summary
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
Andrew Owens, Alexei A. Efros
TL;DR
Audio-visual events can reveal a common underlying event, but traditional computer vision often avoids non-visual modalities. The paper learns a fused representation by predicting temporal synchronization between audio and video, and applies it across localization, recognition, and source separation. The representation supports these downstream tasks, while source-separation metrics do not fully capture predicted-sound quality.
Problem
Traditional computer vision often avoids non-visual modalities, despite temporal cross-modal co-occurrences providing a learning signal for multimodal structure.
Method
A neural network learns a fused audio-visual representation by distinguishing synchronized video clips from clips whose audio is shifted by a few seconds.
Results
The representation is useful for sound-source localization, visual and audiovisual action recognition, and on/off-screen sound separation.
Takeaways & Limitations
The learned representation provides a shared basis for visualizing sound sources, recognizing actions, and separating on-screen from off-screen audio.
Takeaways & Limitations
Quantitative source-separation metrics are limited in conveying predicted-sound quality and can be sensitive to factors that may not be perceptually important.
Abstract
from arXiv · showhide
The thud of a bouncing ball, the onset of speech as lips open -- when visual and audio events occur together, it suggests that there might be a common, underlying event that produced both signals. In this paper, we argue that the visual and audio components of a video signal should be modeled jointly using a fused multisensory representation. We propose to learn such a representation in a self-supervised way, by training a neural network to predict whether video frames and audio are temporally aligned. We use this learned representation for three applications: (a) sound source localization, i.e. visualizing the source of sound in a video; (b) audio-visual action recognition; and (c) on/off-screen audio source separation, e.g. removing the off-screen translator's voice from a foreign official's speech. Code, models, and video results are available on our webpage: http://andrewowens.com/multisensory
1 Introduction
The paper learns a fused audio-visual representation without manual labels by detecting temporal misalignment, then applies it to sound localization, action recognition, and on/off-screen separation.
- 1 Introduction: Self-supervised learning detects whether raw audio and video streams are temporally aligned, forcing the model to fuse visual motion with sound.Training uses synthetically shifted videos rather than manually labeled associations.
- 1 Introduction: The learned representation supports sound-source visualization and visual or audiovisual action recognition.Examples include visualizing an axe impact, mouth opening, or musician’s hands, and classifying a chef chopping an onion.
- 1 Introduction: The model also separates on-screen from off-screen sounds by conditioning source separation on visually masked speakers.The predicted audio is intended to isolate the visible speaker’s voice.
- 1 Introduction: The paper evaluates the representation qualitatively through sound-source visualization and quantitatively on action recognition, while proposing video-conditional source separation for real-world footage.The authors identify television broadcasts as an example of real-world video.
2 Related work
Prior work connects vision and hearing through multisensory integration and audio-visual scene analysis, while earlier self-supervised and separation methods were limited in their inputs, domains, assumptions, or scale.
- 2 Related work: Psychophysical evidence describes vision and hearing as intertwined, including the McGurk effect and sound-timing influences on visual collision judgments.These findings motivate modeling the modalities jointly.
- 2 Related work: Audio-visual scene analysis refers to multisensory perceptual grouping in which audio and visual signals are fused and used jointly.The paper treats its self-supervised representation as a computational model inspired by this process.
- 2 Related work: Earlier self-supervised methods learned correspondences from static images and ambient sound or modeled motion only in limited domains.One cited method modeled drumstick impacts specifically.
- 2 Related work: Existing source-separation approaches made strong sound-motion assumptions, mostly used lab-recorded video, or were tested at relatively small scale.One learning-based on-screen speech model was tested on videos from one speaker.
- 2 Related work: Concurrent methods separated speech using known speaker identities or face detection and tracking to group speakers.These approaches addressed related source-separation and localization tasks with different assumptions.
3 Learning a self-supervised multisensory representation
The paper learns a fused audio-visual representation without manual labels by predicting whether video and audio streams are temporally aligned. An early-fusion multisensory network is trained on synthetically shifted audio and evaluated on alignment and visualization tasks.
- Self-supervised learning: Self-supervision trains the representation to detect whether video frames and audio are temporally synchronized.The model learns audio-visual associations from aligned and misaligned examples rather than manually labeled correspondences.
- Self-supervised learning: The alignment objective maximizes the log-likelihood of distinguishing synchronized streams from versions with randomly shifted audio.The shifted audio is treated as a noisy or misaligned example in the training objective.
- Network design: Early fusion combines audio and visual processing in a 3D multisensory CNN because synchronization detection requires low-level cross-modal integration.The network aligns audio and video activations before concatenating them and applying subsequent 3D convolutions.
- Training setup: Training uses approximately 750,000 AudioSet videos, with 4.2-second clips and audio shifts of 2.0 to 5.8 seconds.The clips contain full-frame-rate video and stereo audio sampled from longer videos.
- Alignment results: 59.9% accuracy on held-out videos exceeds the 50% chance level for the alignment task, while human participants reached 66.6% ± 2.4%.The task is difficult partly because many in-the-wild sounds occur off-screen.
- Representation analysis: Class activation maps visualize the learned attention on held-out videos, with strongest and weakest responses shown for comparison.The strongest-response visualization uses per-image scaling, while Figure 4 presents the weakest responses relative to Figure 3.
4 Visualizing the locations of sound sources
The model visualizes likely sound sources by using learned audio-visual alignment information to generate class activation maps. Its responses are selective to faces, moving mouths, and other salient audiovisual motion, though “sound source” remains operationally defined as motion correlated with sound production.
- Kinetics-Sounds: Kinetics-Sounds examples show strongest CAM responses after removing frames whose activation was only to a face, without labeled training data or per-image heat-map rescaling.The visualization uses a consistent heat-map range across examples.
- Interpretation: For in-the-wild videos, “sound source” denotes motion correlated with sound production rather than one uniquely identifiable physical source.A musician’s lips, larynx, or tuba could each plausibly receive that label.
- Localization method: Class activation maps estimate which space-time patches provide the most information about whether audio and video are aligned.The model scores a patch by the magnitude of its final prediction, |w⊤f(Ix, Ax)|.
- Visualizations: The strongest responses selectively highlight faces and moving mouths, while non-face responses often correspond to unusual salient audiovisual stimuli.Examples include strobe lights and music; weakest-response videos contain relatively few faces.
- Visualizations: The model’s attention varies with on-screen motion, unlike single-frame methods that largely attend to sound-making objects rather than actions.The comparison is qualitative and based on CAM visualizations and supplementary videos.
5 Action recognition
The authors fine-tune the alignment-trained multisensory model for UCF-101 action recognition and compare it with self-supervised and supervised-pretraining baselines. It substantially outperforms earlier self-supervised approaches, approaches visual-only ImageNet-pretrained I3D, and depends partly on audio.
- Setup: The alignment-trained multisensory representation is fine-tuned for action recognition on UCF-101 and compared with unsupervised-learning and 3D CNN methods.The model is initialized with weights learned from the audio-visual alignment task.
- Results: 82.1% accuracy versus 50.9% for Shuffle-and-Learn and 60.3% for O3N shows substantially better performance than prior self-supervised approaches.The authors attribute part of the gap to those methods using single frames or short sequences and tasks requiring less motion analysis.
- Results: The model reaches 84.2%-comparable performance to visual-only ImageNet-pretrained I3D, while remaining below Kinetics-pretrained I3D at 94.5%.The comparison concerns the model using both sound and vision.
- Ablation: Randomly pairing audio and visual streams for self-supervision performs significantly worse than the proposed task, reaching 78.7%.The authors suggest the alternative task can largely be solved without analyzing motion.
- Ablation: Ablating the audio subnetwork causes a 5% performance drop, indicating that sound contributes while visual features remain useful in isolation.This is reported as an ablation-based interpretation of the model’s performance.
6 On/off-screen audio-visual source separation
The paper adapts its learned multisensory representation to separate on- and off-screen sounds from mixed audio, using visual information alongside spectrogram features. Experiments compare the method with audio-only and audiovisual baselines, ablations, transfers, and qualitative real-world examples.
- Source separation model: The model separates synthetic mixtures by predicting on- and off-screen spectrograms from the mixed audio and video.It combines multisensory features with a spectrogram U-net and inverts the predictions to obtain waveforms.
- Source separation model: The system uses raw video without preprocessing such as face detection, while incorporating multisensory features at multiple temporal scales.Video features are temporally aligned, spatially mean-pooled, tiled over frequency, and concatenated with U-net encoder features.
- Evaluation: The evaluation uses VoxCeleb speech mixtures, including gender-based subsets, and transfers models to the simple GRID dataset using On/off and blind-separation metrics.On/off error is measured by ℓ1 distance to true log-spectrograms, while blind-separation metrics are higher-is-better.
- Ablations: Removing temporal motion information reduced performance from 14.8 to 11.4 loss, with a particularly large drop for same-gender mixtures.This tests whether performance depends on motion cues rather than only speaker properties.
- Ablations: Mono conversion did not significantly affect performance, whereas audio-only methods performed better on blind-separation metrics after transfer to GRID, likely because of domain shift.The authors note that 39% of internet-video audio tracks were already mono.
- Evaluation: The audio-visual model outperformed the compared separation methods overall, except that the u-net PIT model was slightly better on SAR.The audio-visual advantage was especially pronounced when both speakers had the same gender.
- Qualitative results: The authors caution that quantitative metrics incompletely capture predicted-sound quality and can be sensitive to perceptually unimportant frequency-representation factors.They therefore supplement metrics with qualitative examples, including mixtures involving multiple speakers and a televised translator.
7 Discussion
The discussion presents the learned temporal multisensory representation as useful across three downstream tasks and identifies broader directions for multisensory learning and audio-visual applications.
- Discussion: Experiments showed the representation was useful for action-recognition pretraining, sound-source visualization, and on/off-screen source separation.These applications span both visual and audio-visual understanding tasks.
- Discussion: Future work could explore additional learning signals beyond temporal misalignment and apply fused audio-visual representations to more audio-understanding and visual tasks.The paper specifically mentions ambient sound as another possible learning signal.
A1 Optimization
The appendix specifies separate optimization procedures for temporal alignment and source separation, including different optimizers, batch sizes, iteration counts, and learning-rate schedules.
- Temporal alignment: Temporal-alignment training used SGD with momentum, batch size 15 across 3 GPUs, and 650,000 iterations.The learning rate began at 0.01 and was halved every 200,000 iterations.
- Source separation: Source-separation fine-tuning used Adam with learning rate 10^-4, batch size 18 across 3 GPUs, and 160,000 iterations.The learning rate dropped by a factor of 10 after 120,000 iterations.
- Source separation: The short-sequence source-separation model was trained for 250,000 iterations, with its learning rate reduced after 150,000 iterations.This procedure used 200 ms sequences.
A2 Baseline model details
The spectrogram-based baseline replaces the waveform audio subnetwork with a 2D CNN built from a truncated ResNet-18, while the appendix also reports classwise synchronization accuracy on Kinetics.
- Baseline model details: The spectrogram variant replaces the audio subnetwork with a 2D CNN adapted from ResNet-18 and truncated after the conv3_1 layer.Temporal and frequency subsampling are applied at specified convolutional layers.
- Baseline model details: The spectrogram input uses log-spectrograms with 8 ms steps, 24 ms windows, and 257 frequency bins.A 3 × F convolution with 512 channels is applied before fusion.
- Synchronization evaluation: Figure A1 reports synchronization-prediction accuracy for Kinetics classes against a 50% chance level.The figure is organized by dataset class rather than by source-separation metric.