PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Visually Aligned Sound from Videos

Peihao Chen, Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, Chuang Gan

arXiv:2008.00820v1cs.CVcs.SDeess.AS

TL;DR

Generating sound from video is difficult because visually irrelevant audio cannot be inferred from visual content and can induce incorrect mappings. REGNET combines appearance and motion features with an audio forwarding regularizer during training, then generates sound from visual features alone at test time. Amazon Mechanical Turk evaluations show improved temporal and content-wise alignment, with generated sound fooling humans at a 68.12% success rate.

Abstract

from arXiv · show

We focus on the task of generating sound from natural videos, and the sound should be both temporally and content-wise aligned with visual signals. This task is extremely challenging because some sounds generated \emph{outside} a camera can not be inferred from video content. The model may be forced to learn an incorrect mapping between visual content and these irrelevant sounds. To address this challenge, we propose a framework named REGNET. In this framework, we first extract appearance and motion features from video frames to better distinguish the object that emits sound from complex background information. We then introduce an innovative audio forwarding regularizer that directly considers the real sound as input and outputs bottlenecked sound features. Using both visual and bottlenecked sound features for sound prediction during training provides stronger supervision for the sound prediction. The audio forwarding regularizer can control the irrelevant sound component and thus prevent the model from learning an incorrect mapping between video frames and sound emitted by the object that is out of the screen. During testing, the audio forwarding regularizer is removed to ensure that REGNET can produce purely aligned sound only from visual features. Extensive evaluations based on Amazon Mechanical Turk demonstrate that our method significantly improves both temporal and content-wise alignment. Remarkably, our generated sound can fool the human with a 68.12% success rate. Code and pre-trained models are publicly available at https://github.com/PeihaoChen/regnet

I. INTRODUCTION

The paper studies generating sound from silent videos, where plausible output requires temporal synchronization and content-wise correspondence despite incomplete visual–audio information. It proposes REGNET, combining appearance and motion features with an audio forwarding regularizer to improve visually aligned sound generation.

Motivation

  • Visually aligned sound requires synchronization with visual events and producing sound types that correspond to the video's content.For example, barking should coincide with a dog's mouth movement and replace unrelated human speech.Source: §I · p. 1
  • REGNET addresses the challenge that visually irrelevant sounds, such as off-camera speech, cannot be inferred from video and can force incorrect visual-to-sound mappings.The resulting information mismatch can impair both temporal and content-wise alignment.Source: §I · p. 1

Method

  • REGNET extracts time-dependent appearance and motion features to provide visual information for predicting temporally and content-wise aligned sound.The model is a spectrogram-based sound generation framework.Source: §I · p. 1
  • The audio forwarding regularizer passes ground-truth sound through a bottleneck and combines its features with visual features during training to control irrelevant sound information.It is removed during testing so sound is generated from visual features alone.Source: §I · pp. 1–2

Evaluation

  • REGNET significantly improves temporal and content-wise alignment over existing methods in evaluations conducted with Amazon Mechanical Turk.The framework also incorporates GANs to improve generated-sound quality and uses a dataset with distinctive video and sound events.Source: §I · p. 2

II. RELATED WORKS

Prior work generates sound from video but does not address the mismatch between visually inferable and irrelevant sound. REGNET formulates this challenge and uses an audio forwarding regularizer so testing can generate visually relevant sound from visual signals alone.

II. RELATED WORKS

  • Existing video-to-sound methods overlook information mismatch, which can cause poor temporal synchronization and content-wise alignment.Source: §II · p. 2
  • The framework is motivated by prior visual-sound learning, text-to-speech, and GAN-based audio generation, while applying GANs to spectrograms for sound synthesis.Source: §II · p. 2

A. Problem formulation

  • Sound is modeled as visually relevant and irrelevant components, with the relevant component related to video while irrelevant sound is independent of visual features and relevant sound.Source: A. Problem formulation · pp. 2–3
  • Because training observes only mixed sound, visual-only models are forced to predict irrelevant sound and may learn an incorrect mapping that produces artifacts and misalignment.Source: A. Problem formulation · p. 3

III. REGNET FRAMEWORK

  • REGNET combines visual encoding, audio forwarding, and generation so the training process can address irrelevant sound while testing removes the forwarding signal.Source: A. Problem formulation · p. 3

B. REGNET framework with audio forwarding regularizer

REGNET uses an audio forwarding regularizer to provide bottlenecked sound information during training alongside visual features. At testing, it removes ground-truth-dependent forwarding and generates sound from visual features alone.

B. REGNET framework with audio forwarding regularizer

C. Why does REGNET work?

REGNET works by allocating visually irrelevant sound to a capacity-limited audio channel while preserving visually relevant sound for the visual features. At test time, removing this channel leaves generation driven by video content.

C. Why does REGNET work?

  • A capacity-limited audio forwarding channel predicts visually irrelevant sound while visual features learn the relevant sound component.The channel must be tuned so it transmits irrelevant information without replacing the visual signal.Source: C. Why does REGNET work? · p. 4
  • If the channel has excessive capacity, it can transmit all sound information and make prediction independent of visual features.Source: C. Why does REGNET work? · p. 4
  • If the channel has insufficient capacity, missing irrelevant-sound information forces the model to learn an incorrect visual-to-sound mapping, producing artifacts and misalignment.Source: C. Why does REGNET work? · p. 4
  • With the right capacity, the channel carries irrelevant sound and the visual features learn a solid mapping to relevant sound.Source: C. Why does REGNET work? · p. 4
  • During testing, REGNET removes the audio forwarding regularizer, so it generates aligned sound from video without inferring visually irrelevant sound.Source: C. Why does REGNET work? · p. 4
  • The central design requirement is proper control of the audio forwarding regularizer’s information capacity.Source: C. Why does REGNET work? · p. 4

IV. REGNET ARCHITECTURE

REGNET combines a time-dependent visual encoder, an audio forwarding regularizer, and a spectrogram generator, with adversarial discrimination and waveform synthesis completing the architecture.

IV. REGNET ARCHITECTURE

  • The visual encoder extracts RGB and flow features at multiple time steps and processes them with convolutions and a bidirectional LSTM.Its outputs are time-dependent encoded visual features for sound generation.Source: §IV · p. 4
  • The audio forwarding regularizer bottlenecks the ground-truth spectrogram, controlling information capacity through output dimension and downsampling rate.It downsamples and then replicates the bottlenecked representation to match the visual features’ temporal dimension.Source: §IV · p. 4
  • During training, the generator combines encoded visual features with bottlenecked real-sound information to reconstruct the real spectrogram.Source: Figure 3 · p. 5
  • The generator predicts an initial spectrogram and adds residual fine structure through a post-network, while applying supervision to both outputs.The final spectrogram is represented as ˆS = ˆI + ˆR.Source: §IV · p. 5
  • A discriminator compares frame features and spectrograms to distinguish real from generated audio and preserve local high-frequency structure.Source: §IV · p. 5
  • WaveNet converts synthesized spectrograms into 16-bit waveforms using a 10-component mixture of logistic distributions at 22,050 Hz.Source: §IV · p. 5

V. EXPERIMENTS

The experiments describe dataset preparation, alignment and reality comparisons against a state-of-the-art method, and ablations of the audio forwarding regularizer.

V. EXPERIMENTS

  • The experimental section covers datasets, comparison with a state-of-the-art method, and ablation studies of the audio forwarding regularizer.Source: §V · p. 5

A. Datasets

The evaluation uses synchronized sound classes from VEGAS and AudioSet, with human judgments comparing alignment and visualizations of three misalignment types.

A. Datasets

  • The datasets include four sound classes from VEGAS and four from AudioSet selected for high visual-audio synchronization.They are grouped as Sub-VEGAS and Sub-AudioSet, with about 1,626 videos per class on average.Source: A. Datasets · p. 5
  • The alignment evaluation measures Sound-Missing, Sound-Redundant, and Sound-Mismatched cases in human judgments.Source: Figure 4 · p. 6
  • Table I reports the percentage of human judgments preferring one method over another based on three alignment criteria.Source: Table I · p. 6

B. Implementation details

REGNET is evaluated using standardized spectrogram preprocessing, training settings, and human judgments of temporal and content-wise alignment against Visual2Sound. On Sub-VEGAS, it improves all three alignment measures and qualitative cases show fewer missing, redundant, and mismatched sounds.

B. Implementation details

  • The evaluation uses 21.5-fps videos padded to 10 seconds, 22,050-Hz audio, and 80-bin mel spectrograms with 860 time steps.Training lasts 1,000 epochs with Adam; the audio forwarding regularizer uses dimension D=32 and downsampling rate S=860.Source: B. Implementation details · p. 6
  • Visual2Sound serves as the state-of-the-art baseline and provides the comparison outputs used in evaluation.It uses SampleRNN to generate waveforms from encoded visual features.Source: B. Implementation details · p. 6

C. Evaluation on alignment

  • The AMT alignment study compares REGNET and Visual2Sound on paired videos using Sound-Missing, Sound-Redundant, and Sound-Mismatched judgments.The first two metrics assess temporal alignment, while Sound-Mismatched assesses content-wise alignment.Source: C. Evaluation on alignment · p. 6, B. Implementation details · p. 6
  • REGNET outperforms Visual2Sound by 21.24% on Sound-Missing, 18.66% on Sound-Redundant, and 13.54% on Sound-Mismatched.Approximately 60% of participants preferred REGNET for the first two temporal-alignment measures.Source: C. Evaluation on alignment · p. 7
  • Qualitative examples show REGNET recovering fireworks sound, suppressing redundant dog sounds, and avoiding a metal-hit mismatch in drum videos.The corresponding baseline outputs miss, add, or misclassify sound events.Source: C. Evaluation on alignment · p. 7

D. Evaluation on real-or-fake task

The real-or-fake evaluation tests whether generated sounds are judged real, while a simple action-detection-and-clip method provides an additional comparison. REGNET reaches a high human-fooling rate and substantially exceeds the simple method on dog videos.

D. Evaluation on real-or-fake task

  • 68% of REGNET-generated sounds fool humans into judging them real, close to the success rate of real sounds.The task combines videos with either generated or real sound and asks turkers to judge general alignment and sound quality.Source: D. Evaluation on real-or-fake task · p. 7

E. Comparison with a simple method

  • The simple method detects action occurrences from sound and optical-flow amplitudes, then plays manually cropped training clips of suitable lengths.It uses a visual encoder and multilayer perceptron for binary action classification.Source: E. Comparison with a simple method · p. 7
  • 71.09% of REGNET-generated dog sounds fool humans, compared with 43.65% for the simple method.The simple method struggles with diverse sounds within one video and requires difficult action-occurrence thresholds without annotations.Source: E. Comparison with a simple method · p. 7

F. Effectiveness of audio forwarding regularizer

Simulated irrelevant sounds test whether the audio forwarding regularizer preserves visually relevant audio while suppressing nuisance components. REGNET consistently outperforms its no-regularizer counterpart in objective and human evaluations.

F. Effectiveness of audio forwarding regularizer

  • The simulated Dog-gauss and Dog-fireworks datasets inject Gaussian noise or fireworks audio into corresponding dog recordings as visually irrelevant sound.The original audio is treated as the visually relevant ground truth.Source: F. Effectiveness of audio forwarding… · p. 8
  • REGNET and Ours-VS achieve lower L2 distances than Ours-woREG on both simulated datasets: 1.96, 2.03 vs. 3.63 and 1.87, 1.82 vs. 3.02.The results support the regularizer’s role in generating visually relevant sound and show that explicit visual input is unnecessary for the regularizer.Source: F. Effectiveness of audio forwarding… · p. 8, Table VI · p. 8
  • Without the regularizer, the model learns mappings from visual features to irrelevant noise, whereas REGNET focuses on the visually relevant component.Spectrogram visualizations provide the qualitative comparison.Source: F. Effectiveness of audio forwarding… · p. 8
  • REGNET improves real-or-fake performance over Ours-woREG by 2.34%, 9.40%, and 5.10% across three training datasets.Gaussian and fireworks interference reduce the counterpart’s performance by 10.45% and 5.05%, while REGNET remains robust.Source: F. Effectiveness of audio forwarding… · p. 8

G. Exploration on regularizer output

Additional experiments probe what information the audio forwarding regularizer retains and whether that information can be reconstructed or injected into generated sound. The results indicate that its representation primarily carries visually irrelevant background audio.

G. Exploration on regularizer output

  • The exploration uses Dog-gauss, Dog-fireworks, Dog-drum, Baby-gauss, and Baby-fireworks simulated datasets, with 128 test videos per type.The experiments separately measure output similarity, reconstructable sound components, and background-sound injection.Source: G. Exploration on regularizer output · pp. 8–9
  • Regularizer outputs are more similar for sounds sharing background audio, indicating that the representation passes through background information while ignoring visually relevant sound.Dog-fireworks and Baby-fireworks produce higher cosine similarity than data with different background sounds.Source: G. Exploration on regularizer output · p. 9
  • With visual features set to zero, generated spectrograms are closer to irrelevant than relevant sounds: 0.84 vs. 4.36, 1.97 vs. 3.69, and 2.29 vs. 4.53.The compared irrelevant sounds are Gaussian, fireworks, and drum audio.Source: G. Exploration on regularizer output · p. 9
  • Replacing the zero vector with regularizer outputs reduces distances to background spectrograms: 2.72 vs. 5.05, 2.78 vs. 5.56, and 2.79 vs. 6.11.This indicates that background sound information can be mixed into the generated output during testing.Source: G. Exploration on regularizer output · p. 9

H. Exploration on REGNET variants

REGNET performs best when the audio forwarding regularizer has controlled capacity, while excessive capacity can replace visual-to-sound learning and insufficient capacity can introduce noise. The discriminator further improves sound realism, supporting the conclusion that REGNET produces more plausible and aligned video-conditioned sound.

H. Exploration on REGNET variants

  • REGNET achieved the lowest L2 distance, 1.96 versus 3.18, 3.19, and 3.08 for narrow and wide variants, showing that controlled regularizer capacity is necessary.The wide variant retained target-spectrogram information during training instead of learning visual-to-sound mappings, whereas the narrow variant produced noisy spectrograms.Source: H. Exploration on REGNET variants · pp. 9–10
  • 71.09% of REGNET sounds were judged real, exceeding the wide variants’ 62.50% and 66.67% and the narrow variant’s 70.18%.REGNET also achieved the highest scores on all three alignment metrics.Source: H. Exploration on REGNET variants · p. 10
  • The wide variant’s low Sound-Missing score indicates that excessive audio-forwarding capacity can prevent learning the visual-to-sound mapping needed when actions occur.Its regularizer can provide sufficient target-spectrogram information during training, encouraging reconstruction rather than visual correspondence learning.Source: H. Exploration on REGNET variants · pp. 9–10

I. Exploration on discriminator

  • Removing adversarial loss reduced the human fooling rate from REGNET’s 68.12% to 64.04%, indicating that the discriminator improves generated-audio fidelity.Source: §I · p. 10

VI. CONCLUSION

  • The audio forwarding regularizer supplies training-time information that helps REGNET learn concrete visual–sound correspondence, while testing uses visual features to generate aligned sound.Source: §VI · p. 10

Related papers

Loading 2008.00820v1…