Source-linked AI summary
Visual to Sound: Generating Natural Sound for Videos in the Wild
Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, Tamara L. Berg
TL;DR
The paper asks whether models can generate natural sound from video content, including for virtual reality and accessibility. It trains waveform-generating models on cleaned wild videos and finds realistic, temporally synchronized audio, with over 70% of generated sounds fooling human listeners.
Problem
The paper addresses the problem of generating natural sound from video content, a capability relevant to virtual reality, automatic sound editing, and accessibility.
Method
The authors clean an AudioSet-derived dataset and train video-to-waveform models using a video encoder and SampleRNN-based sound generator.
Results
Over 70% of generated sounds fooled humans into thinking they were real, while the flow-based method achieved the best category- and instance-level retrieval accuracy.
Takeaways & Limitations
The results support generating realistic sound from videos in the wild with temporal synchronization between visual inputs and generated audio.
Takeaways & Limitations
The work does not explicitly handle high-level visual reasoning during sound prediction and assumes training video and sound are directly related most of the time.
Abstract
from arXiv · showhide
As two of the five traditional human senses (sight, hearing, taste, smell, and touch), vision and sound are basic sources through which humans understand the world. Often correlated during natural events, these two modalities combine to jointly affect human perception. In this paper, we pose the task of generating sound given visual input. Such capabilities could help enable applications in virtual reality (generating sound for virtual scenes automatically) or provide additional accessibility to images or videos for people with visual impairments. As a first step in this direction, we apply learning-based methods to generate raw waveform samples given input video frames. We evaluate our models on a dataset of videos containing a variety of sounds (such as ambient sounds and sounds from people/animals). Our experiments show that the generated sounds are fairly realistic and have good temporal synchronization with the visual inputs.
1. Introduction
The paper asks whether models can generate natural sound from video content, targeting applications including virtual reality, automated sound editing, and accessibility. It takes a first step by predicting raw audio from videos in the wild and introduces a cleaned dataset and evaluations.
- Motivation: The paper explores whether computational models can learn the relationship between visual content and sound for several practical applications.Motivations include immersive virtual reality, automatic sound effects, and associating sound with visual information for people with visual impairments.
- Problem scope: The initial task is generating audio for video based on viewable content rather than performing high-level visual reasoning.
- Approach: Models directly predict raw waveform samples from input videos to learn associations between generated sounds and visual scenes or object interactions.The framework combines a video encoder with a sound generator and includes three visual-information encoding variants.
- Setting: The work addresses natural sound from videos collected in the wild, whereas cited prior work used experimental settings such as indoor recordings or hitting sounds.
- Dataset: The authors clean an AudioSet subset by verifying target objects in video and target sounds in audio at 2-second intervals.The cleaning targets human, animal, and other natural sounds so the data better supports learning the video-to-audio mapping.
- Contribution: The released dataset contains 28,109 cleaned videos totaling 55 hours across 10 object categories, alongside numerical and human evaluations.
2. Related work
Prior work uses synchronized video and sound mainly for self-supervised representation learning or constrained audio-generation tasks. This paper is positioned against those settings by targeting more diverse real-world scenarios.
- Video and sound self-supervision: Video-and-sound self-supervision uses their concurrent occurrence in unlabeled videos to learn correspondence, semantics, similarity, or shared representations.
- Speech synthesis: Speech-synthesis work includes WaveNet and SampleRNN, with SampleRNN using a hierarchical recurrent structure for sequential audio generation.
- Mapping visual to sound: Prior mapping methods directly generate audio from visual inputs, including hitting sounds conditioned on materials and interactions and image-to-sound generation with GANs.
- Gap: The paper extends visual-to-sound generation toward more diverse and challenging real-world scenarios.
3. Visually Engaged and Grounded AudioSet (VEGAS)
VEGAS is a cleaned AudioSet-derived dataset designed for video-to-sound generation, with verified visual and auditory content across natural-sound categories. Its clips are labeled in short segments, filtered, and merged into videos of varied lengths.
- Motivation: VEGAS addresses noisy AudioSet data, where videos and audios may be loosely related, target sounds masked by music, or clips misclassified.Baseline models trained on original data produced sounds accompanied by noise such as chaotic human chatting.
- Data collection: The dataset covers 10 categories including human, animal, and ambient sounds, with up to 3,000 randomly selected videos per category when more data was available.
- Data collection: Annotators verify the target object or event in both video and audio, labeling clips separately in 2-second segments.Each segment receives annotations from three turkers, and clips labeled No in either modality are removed.
- Data visualization: Figure 1 aligns colored frame borders with waveform marks to indicate each frame’s position within its video.
- Data processing: Adjacent verified clips labeled Yes or Sort of are merged into videos ranging from 2–10 seconds, retaining additional variation.
- Data statistics: 28,109 videos remain after annotating 132,209 clips and removing 34,392, averaging 7 seconds and totaling 55 hours.The dataset spans 10 categories, and most videos are longer than 8 seconds.
4. Approaches
The paper formulates video-to-sound generation as conditional synthesis of quantized raw waveforms, using an encoder-decoder framework with hierarchical recurrent sound generation. It evaluates frame-to-frame, sequence-to-sequence, and flow-based visual encoders, including optical-flow features to capture motion and synchronization.
- Problem formulation: The model estimates the conditional probability of waveform values given video-frame representations, with video sequences typically much shorter than audio sequences.Waveform samples are rescaled and linearly quantized into 256 bins, while the audio sampling rate makes n much larger than m.
- Sound generator: The overall architecture combines a video encoder with a hierarchical recurrent sound generator based on SampleRNN.The generator uses coarse-to-fine recurrent tiers to handle extremely long sequences and capture dependencies between distant samples.
- Sound generator: SampleRNN predicts quantized waveform samples through a fine-tier MLP conditioned on previous samples and outputs from coarser recurrent tiers.The finest tier performs 256-way classification, while coarser tiers jointly predict multiple waveform samples at each timestep.
- Frame-to-frame method: The frame-to-frame method concatenates each VGG19 fc6 frame representation with coarsest-tier sound-generator nodes after duplicating visual features to match audio and video sequence lengths.The duplication factor is s = ceiling[sraudio/srvideo], maintaining cross-modal alignment despite different sampling rates.
- Sequence-to-sequence method: The sequence-to-sequence method recurrently encodes video features and uses the final encoder hidden state to initialize the sound generator's coarsest-tier RNN.Unlike the frame-based model, it expects the model to learn video-audio alignment through encoding and decoding rather than enforcing it explicitly.
- Architecture overview: Figure 3 summarizes three conditioning structures: direct frame-to-frame concatenation, recurrent encoding with hidden-state initialization, and the same recurrent structure for flow-based features.The generator's integer outputs are linearly mapped from [0, 255] to raw waveforms in [-1, 1].
- Flow-based method: The flow-based method concatenates VGG visual features with optical-flow features so the encoder can explicitly represent object motion relevant to sound timing.Its architecture otherwise follows the sequence-to-sequence design, recurrently encoding the combined features before decoding with SampleRNN.
5. Experiments
The experiments evaluate three video-to-audio models through qualitative visualization, numerical metrics, retrieval, and human judgments. Flow and Seq generally outperform Frame, with Flow strongest overall and especially helpful for synchronization-sensitive categories.
- Model and training details: The study trains Frame, Seq, and Flow models independently for each of 10 categories, using padded 10-second videos and a separate multi-category experiment.All models use a 3-tier SampleRNN sound generator; the multi-category model combines data from all categories.
- Qualitative visualization: Waveform visualizations show that generated audio can reflect scene events, including plausible extra fireworks explosions and train-volume changes as the object approaches.All three models implicitly learn the distance-related amplitude effect in rail transport videos.
- Numerical evaluation: Flow and Seq achieve lower training and testing cross-entropy loss than Frame, with Seq best on training loss and Flow best on testing loss.The comparison is reported for the average cross-entropy loss of the sound generator.
- Numerical evaluation: All methods outperform chance in category- and instance-level audio retrieval, while Flow achieves the best accuracy under both top-1 and top-5 metrics.Chance is 10% for category retrieval and 0.78% for instance retrieval.
- Human evaluation experiments: Human forced-choice evaluations favor Seq and Flow over Frame, with Flow performing best overall and gaining its largest advantage on synchronization-sensitive categories.The strongest category-level advantages occur for Fireworks and Drum.
- Human evaluation experiments: Generated audio achieves reasonable visual discrimination for Dog and Fireworks, whereas ambient categories are challenging for both generated and real audio.Dog and Fireworks substantially exceed the 50% chance level in the matching task.
- Additional experiments: The multi-category model fools humans on 46.29% of generated sounds on average, below the 73.63% achieved by the best single-category model.This result comes from the real/fake experiment on the VEGAS dataset.
6. Conclusion
The paper introduces video-to-sound generation in the wild, supported by a cleaned dataset and three deep network variants. Evaluations find that over 70% of generated sounds can fool humans into judging them real.
- The work introduces realistic sound generation from videos collected in the wild.
- The authors create an AudioSet-derived dataset and train three visual-to-sound deep network variants.
- Over 70% of generated sounds fool humans into thinking they are real.