PaperlayerView exact revision on arXiv

Source-linked AI summary

2.5D Visual Sound

Ruohan Gao, Kristen Grauman

arXiv:1812.04204v4cs.CV

TL;DR

Binaural recordings are scarce and difficult to obtain, motivating recovery of spatial sound from common monaural audio. The paper combines audio with video in MONO2BINAURAL to predict binaural audio, then uses the learned representation for source separation. It reports state-of-the-art audio spatialization and improved audio-visual source separation, while noting failures with visually similar multiple objects.

  • Problem

    Binaural recordings are scarce and require expensive equipment and expertise, while monaural audio lacks source-location information.

  • Method

    MONO2BINAURAL combines monaural audio with accompanying video frames to predict binaural audio using visual spatial cues.

  • Results

    The framework achieves state-of-the-art audio spatialization and improves audio-visual source separation using predicted binaural audio as a representation.

  • Takeaways & Limitations

    The approach can generate spatialized sound for off-the-shelf video and provide a self-supervised signal for audio-visual source separation.

  • Takeaways & Limitations

    Spatialization can fail when multiple objects have similar appearance, although motion-based instance localization is suggested as future work.

Abstract

from arXiv · show

Binaural audio provides a listener with 3D sound sensation, allowing a rich perceptual experience of the scene. However, binaural recordings are scarcely available and require nontrivial expertise and equipment to obtain. We propose to convert common monaural audio into binaural audio by leveraging video. The key idea is that visual frames reveal significant spatial cues that, while explicitly lacking in the accompanying single-channel audio, are strongly linked to it. Our multi-modal approach recovers this link from unlabeled video. We devise a deep convolutional neural network that learns to decode the monaural (single-channel) soundtrack into its binaural counterpart by injecting visual information about object and scene configurations. We call the resulting output 2.5D visual sound---the visual stream helps "lift" the flat single channel audio into spatialized sound. In addition to sound generation, we show the self-supervised representation learned by our network benefits audio-visual source separation. Our video results: http://vision.cs.utexas.edu/projects/2.5D_visual_sound/

1. Introduction

Binaural recordings provide spatial sound but are difficult to obtain, while monaural audio loses source locations. The paper uses accompanying video to recover spatial cues, generate binaural audio, and support audio-visual source separation.

  • Binaural recordings are difficult to obtain because they require expensive equipment and specialized expertise.
  • Monaural audio collapses independent sound streams to one spatial point, preventing listeners from sensing source locations.
  • Video frames reveal object and scene configurations that can spatialize otherwise non-spatial monaural audio.For example, visual positions can indicate separate sound sources such as musicians on the left and right.
  • MONO2BINAURAL is an encoder-decoder convolutional network that combines mixed single-channel audio with video frames to predict spatially aligned binaural audio.
  • The conversion process also provides a self-supervised representation that benefits audio-visual source separation without semantic annotations or single-source data preparation.
  • The paper introduces FAIR-Play, a 5.2 hour binaural video dataset, and validates the approach on four datasets spanning instruments, street scenes, travel, and sports.

2. Related Work

Prior work explores audio generation, spatialization, source separation, and self-supervised audio-visual learning. This paper differs by spatializing ordinary monaural audio with normal-field-of-view video and using the result for separation.

  • Earlier systems generate sounds from silent video using material cues, recurrent networks, conditional GANs, or 3D-shape simulators.
  • Prior video-based spatialization either requires a room-specific acoustic impulse response or generates ambisonics from 360° video and mono audio.
  • Audio-only separation becomes easier with multiple microphones or binaural audio, motivating conversion from mono to binaural using video.
  • Audio-visual separation research spans classical signal-processing methods, deep learning for multiple sound categories, and emerging on- and offscreen tasks.
  • The paper frames MONO2BINAURAL as self-supervised learning that uses visual frames to supervise audio spatialization while learning audio representations.

3. Approach

The approach collects synchronized video and binaural audio, formulates mono-to-binaural prediction using visual supervision, and extends the learned representation to audio-visual source separation.

  • 3.1. FAIR-Play Data Collection: FAIR-Play collects synchronized video and binaural audio with a custom rig combining a binaural microphone, camera, and recorder.The rig records in a music room containing varied instruments and performance configurations.
  • 3.2. Mono2Binaural Formulation: The mono-to-binaural task mixes left and right channels into monaural audio, then uses accompanying video to address the resulting under-constrained channel-recovery problem.Visual spatial information provides supervision for recovering separate binaural channels.
  • 3.3. Mono2Binaural Network: MONO2BINAURAL takes monaural audio and a visual frame, extracts audio-visual features with U-NET and ResNet-18, and predicts a complex mask for the audio difference signal.The predicted difference signal is combined with the mono input to reconstruct the left and right channels.
  • 3.3. Mono2Binaural Network: Predicting the channel difference rather than directly predicting both channels discourages the network from copying the input and encourages visual reasoning about binaural differences.The difference signal is estimated in the frequency domain using a complex spectrogram and mask, then converted back with ISTFT.
  • 3.4. Audio-Visual Source Separation: For audio-visual source separation, the method mixes predicted binaural tracks from two videos and learns to separate each sound conditioned on its corresponding visual frame.This testbed follows the Mix-and-Separate framework and uses the same base architecture with paired video inputs.

4. Experiments

The experiments evaluate binaural generation and source separation across diverse datasets using quantitative metrics, baselines, user studies, and qualitative analyses. MONO2BINAURAL consistently outperforms the tested generation baselines, while visual ambiguity and difficult scenes remain failure cases.

  • Experimental Setup: The evaluation spans four datasets covering musical instruments, street scenes, travel, and sports, with FAIR-Play providing professionally recorded binaural audio.FAIR-Play contains 1,871 ten-second clips and is split into 10 random train/validation/test splits; the other datasets are decoded from ambisonics using an HRTF.
  • Binaural Generation Accuracy: The study compares MONO2BINAURAL against Ambisonics, Audio-Only, Flipped-Visual, and Mono-Mono baselines using STFT and envelope distances, for which lower is better.The Audio-Only baseline removes the visual stream, Flipped-Visual supplies incorrect visual information, and Mono-Mono copies the mono signal to both channels.
  • Binaural Generation Accuracy: MONO2BINAURAL outperforms all baselines consistently on all four datasets, with the largest advantage over Audio-Only when sound sources are not centered.Flipped-Visual performs much worse, supporting that the network learns to localize sound sources using the accompanying visual frames.
  • User Studies: In user studies with 18 participants, predicted binaural audio produced a more realistic 3D sensation and more accurate perceived directionality than the baselines.The studies used 45 comparison pairs across datasets and ten FAIR-Play instrument clips; the second study used real mono audio recorded by a GoPro microphone.
  • Source Localization and Failure Cases: Qualitative visualizations show attention regions often correlate with active instruments, moving cars, and activity locations, but similar instruments, silent or noisy scenes, and many sources remain difficult.The system can also fail with multiple visually similar human speakers; incorporating motion is suggested as future work for instance-level spatialization.
  • Audio-Visual Source Separation: The predicted binaural representation also supports audio-visual source separation, where the comparison evaluates SDR, SIR, and SAR in decibels with higher values preferred.The separation comparison holds other network aspects constant so performance differences can be attributed to binauralization self-supervision.

5. Conclusion

The approach converts single-channel audio into binaural audio using visual object and scene configurations, producing more immersive 2.5D visual sound. MONO2BINAURAL also improves audio-visual source separation by providing a more informative audio representation.

  • The method converts single-channel audio into binaural audio by leveraging object and scene configurations in video frames.
  • The predicted 2.5D visual sound offers a more immersive audio experience.
  • MONO2BINAURAL achieves state-of-the-art performance on audio spatialization.
  • Using predicted binaural audio as a better audio representation boosts a modern model for audio-visual source separation.

B. Details of MONO2BINAURAL Network

The MONO2BINAURAL network combines a visual branch based on ImageNet-pretrained ResNet-18 with a U-NET-style audio branch. The audio branch uses encoder-decoder processing and skip connections for its prediction task.

  • The visual branch uses ImageNet-pretrained ResNet-18 to extract a 14x7x512 feature map, then reduces it to 14x7x8 with a 1x1 convolution.
  • The audio branch is a U-NET-style encoder-decoder with five convolution layers, five up-convolution layers, and skip connections.
  • All convolutions and up-convolutions use 4 x 4 filters with stride 2, followed by BatchNorm and ReLU operations.
  • A final Sigmoid layer bounds the complex-mask values, while encoder leaky ReLUs use slope 0.2 and decoder ReLUs are non-leaky.

C. Details of MIX-AND-SEPARATE Network

The MIX-AND-SEPARATE network processes paired training videos with shared-weight visual branches and a mixed-audio branch. It combines visual and audio features to predict a separation mask for each video.

  • The network takes a pair of training video clips and uses two shared-weight visual branches to extract their visual features.
  • Its audio branch processes the mixed audio, combines audio and visual features, and predicts a mask for each video.
  • The method follows prior work by using ratio masks and log-magnitude representations.

D. Implementation Details

Training and evaluation use sampled audio segments paired with video frames, with separate preprocessing and optimization settings for MONO2BINAURAL and audio-visual source separation. User studies evaluate how listeners perceive predicted binaural audio.

  • MONO2BINAURAL training samples 0.63s audio segments, normalizes RMS levels, and forms 257×64×2 complex spectrograms with accompanying center video frames.
  • Figure 7 presents interfaces for two user studies testing listeners’ perception of predicted binaural audio.
  • The MONO2BINAURAL network uses 448×224 crops, color and intensity jittering, Adam optimization, batch size 256, and 1,000 training epochs.
  • Audio-visual source separation samples paired 2.55s segments, mixes them, and forms 257 × 256 log-magnitude spectrograms for each channel.

E. User Study Interface

The figure presents interfaces for two user studies evaluating predicted binaural audio against ground truth and baselines through perceptual judgments.

  • The first study presents participants with a 10-second ground-truth binaural clip and its visual frame before comparison listening.
  • Participants compare binaural audio generated by the method or baselines, selecting which version creates a better 3D sensation matching ground truth.The listed baselines include Ambisonics, Audio-Only, and Mono-Mono.
  • The second study asks participants to identify the direction of a specified instrument after listening to ground-truth or predicted binaural audio.The interface example in Fig. 7b illustrates this directional listening task.
Loading 1812.04204v4…