Source-linked AI summary
Self-Supervised Generation of Spatial Audio for 360 Video
Pedro Morgado, Nuno Vasconcelos, Timothy Langlois, Oliver Wang
TL;DR
Spatial audio is valuable for immersive 360° video, but spatial-audio microphones are uncommon and consumer cameras often provide only mono recordings. The paper introduces a self-supervised, end-to-end system that uses 360° video and mono audio to separate and localize sources, then generate first-order ambisonics. Experiments on two datasets show plausible spatial audio and better performance than a domain-independent baseline, while performance remains limited on challenging scenes.
Problem
Spatial audio is important for immersive 360° video, yet consumer cameras commonly record mono audio and spatial-audio capture is expensive or technically challenging.
Method
The system uses multi-modal audio-visual analysis, source separation, and localization to convert mono audio into first-order ambisonics through self-supervised training.
Results
The proposed neural network generates plausible spatial audio on two datasets and outperforms a domain-independent U-NET baseline.
Takeaways & Limitations
360° video and mono audio can provide sufficient information for inferring spatial sound-source locations and generating spatial audio.
Takeaways & Limitations
Performance degrades with overlapping sources, reverberation, ambiguous visual-to-sound mappings, and the still-low number of available 360° videos with spatial audio.
Abstract
from arXiv · showhide
We introduce an approach to convert mono audio recorded by a 360 video camera into spatial audio, a representation of the distribution of sound over the full viewing sphere. Spatial audio is an important component of immersive 360 video viewing, but spatial audio microphones are still rare in current 360 video production. Our system consists of end-to-end trainable neural networks that separate individual sound sources and localize them on the viewing sphere, conditioned on multi-modal analysis of audio and 360 video frames. We introduce several datasets, including one filmed ourselves, and one collected in-the-wild from YouTube, consisting of 360 videos uploaded with spatial audio. During training, ground-truth spatial audio serves as self-supervision and a mixed down mono track forms the input to our network. Using our approach, we show that it is possible to infer the spatial location of sound sources based only on 360 video and a mono audio track.
1 Introduction
360° video is increasingly accessible, but spatial audio remains uncommon because consumer cameras typically record only mono audio. The paper introduces a self-supervised system that converts mono audio and 360° video into spatial audio, supported by new datasets and evaluation.
- Spatial audio supports full-scene awareness and immersive 360° viewing by matching sound direction to the viewer’s position.
- Less than 5% of a random sample of 1,000 YouTube 360° videos had spatial audio at submission time.
- The proposed 360° spatialization task upconverts a single mono recording into first-order ambisonics guided by full 360° video.
- The system analyzes audio and video, separates sound sources, localizes them, and combines the results to generate spatial audio.
- Two spatial-audio 360° video datasets support training and validation: one recorded in a constrained domain and one collected in the wild from YouTube.
- Ground-truth spatial audio supervises training while a mixed-down mono track is supplied as the network input.
2 Related Work
Prior work spans self-supervised learning, audio generation, source separation, sound localization, and cross-modal audio-visual analysis. This paper situates audio spatialization within these areas while targeting ambisonic generation from non-spatial audio and corresponding 360° video.
- The paper proposes audio spatialization as a problem related to self-supervised learning, audio generation, source separation, and audio-visual cross-modal learning.
- Spatial audio: Ambisonics represents a sound field through spherical harmonics, and its coefficients can be decoded for playback; the paper aims to generate these coefficients from mono audio and video.The representation uses sound direction and time as inputs to a truncated spherical harmonic expansion.
- Self-supervised learning: Self-supervision has been applied to sound-video misalignment detection, audio super-resolution, and learning ambisonic audio from in-the-wild 360° video with spatial audio.
- Generative models and source separation: Recent generative models have produced complex images, text, speech, and music, while data-driven neural methods have advanced audio source separation.
- Source localization: Existing sound-localization methods commonly rely on calibrated microphone arrays, beamforming, or binaural and HRTF cues, limiting their applicability to in-the-wild video.
- Cross visual-audio analysis: Cross-modal audio-visual methods learn audio representations, localize dominant sources, and associate repeated visual motions with corresponding sounds.
3 Method
The method upconverts mono audio from 360° video into FOA by separating sound sources, localizing them using synchronized audio-visual features, and combining separated waveforms with localization weights. It uses self-supervision from paired spatial recordings and evaluates both signal similarity and localization accuracy across two datasets.
- Task definition: 360° spatialization generates ambisonic channels from non-spatial audio and corresponding 360° video, targeting four-channel first-order ambisonics.FOA stores four first-order spherical-harmonic coefficients, represented here as φw, φy, φz, and φx.
- Self-supervision: Under an omnidirectional-microphone assumption, mono audio serves as zeroth-order ambisonics, so the network predicts the missing φx(t), φy(t), and φz(t) channels.The framework can also be adapted to other conversion scenarios by changing the input and output audio tracks.
- Audio-visual analysis: The architecture combines STFT-based audio features with RGB and optical-flow video features, synchronizes their rates, and feeds the joint representation to separation and localization modules.Audio uses 25 ms STFT segments with 25% hop size; video uses a two-stream ResNet-18 encoder for appearance and motion.
- Source separation and localization: The separation module extracts up to k source tracks with a U-Net decoder, while the localization module predicts per-source spherical-harmonic weights from multimodal features.The weights are interpreted as spherical harmonics evaluated at each predicted source position.
- Ambisonic generation: The system generates first-order ambisonic channels by summing each separated waveform multiplied by its localization weights, transforming the mixed input rather than generating audio from scratch.This design is described as enabling fast deployment speeds with high-quality results.
4 Evaluation
The evaluation combines ablations, baselines, qualitative comparisons, real-mono tests, and user studies to assess spatial-audio generation. Results show strong reliance on visual and task-specific architecture, plausible spatial outputs, real-time synthesis, and remaining failures on difficult scenes.
- Real time performance: 1s of spatial audio at 48000Hz is generated in 103ms on a single 12GB Titan Xp GPU.
- Ablations: The network relies heavily on visual features, while the visual encoder and separation block are necessary for complex videos.Removing both visual encoders performs poorly overall; the visual encoder and separation block matter especially for YT-MUSIC and YT-ALL.
- Baselines: Prior spatial audio distributions alone are insufficient for accurate ambisonic conversion.
- Baselines: The proposed architecture significantly outperforms the U-NET baseline, demonstrating the importance of task-specific design for spatial-audio generation.
- Qualitative results: Qualitative outputs have spatial energy distributions similar to ground truth while preserving the sound fidelity of the original mono input.
- User study: Participants labeled the approach's audio as real more often than U-NET and MONO in both studies, with 62% for HMD viewing versus 55% in-browser viewing.The user studies used both browser and HMD experiences; the reported comparison is statistically significant for the approach against U-NET and MONO.
5 Discussion
The approach is an initial solution for generating spatial audio from mono recordings and 360° video, but challenging scenes and limited data constrain its reliability. Future hardware and datasets may extend its relevance and capabilities.
- Limitations: Many overlapping sources, reverberant environments, and visually ambiguous sources can cause incorrect sound-source separation or localization.The authors identify these as challenging examples for general-purpose spatial audio generation.
- Future work: The method remains relevant for legacy mono or stereo recordings and could potentially increase spatial resolution by converting to higher-order ambisonics.The authors specifically mention up-converting first-order ambisonics into second-order ambisonics as a future direction.
- Conclusion: The paper presents an end-to-end trainable approach for up-converting mono recordings into spatial audio from corresponding 360° video.The conclusion describes this as the paper’s first approach and reports that its generator performs substantially better than a domain-independent baseline.