PaperlayerView exact revision on arXiv

Source-linked AI summary

ViSAudio: End-to-End Video-Driven Binaural Spatial Audio Generation

Mengchen Zhang, Qi Chen, Tong Wu, Zihan Liu, Dahua Lin

arXiv:2512.03036v1cs.CVcs.AI

TL;DR

Binaural video-to-audio generation remains limited by mono-first spatialization, despite the value of spatial audio for immersion. This paper introduces BiAudio and ViSAudio for end-to-end binaural generation from silent video, and reports state-of-the-art performance with adaptation to viewpoint changes.

  • Problem

    Video-to-audio research has focused predominantly on mono output, while binaural methods commonly rely on a mono-first pipeline and end-to-end binaural generation from silent video remains unexplored.

    Source: §1 · p. 2

  • Method

    The paper introduces the BiAudio dataset and ViSAudio, which uses conditional flow matching, dual audio branches, and a conditional spacetime module for video-driven binaural generation.

    Source: §1 · p. 2

  • Results

    ViSAudio achieves state-of-the-art performance and generates immersive binaural audio that adapts to viewpoint changes.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The work demonstrates end-to-end binaural audio generation from silent video using a dataset with diverse real-world scenes and camera motions.

    Source: §1 · p. 2, §6 · p. 8

    The model handles short clips, limiting capture of long-range temporal dependencies and complex acoustic events; background noise may introduce artifacts in generated outputs.

    Source: §6 · p. 8

Abstract

from arXiv · show

Despite progress in video-to-audio generation, the field focuses predominantly on mono output, lacking spatial immersion. Existing binaural approaches remain constrained by a two-stage pipeline that first generates mono audio and then performs spatialization, often resulting in error accumulation and spatio-temporal inconsistencies. To address this limitation, we introduce the task of end-to-end binaural spatial audio generation directly from silent video. To support this task, we present the BiAudio dataset, comprising approximately 97K video-binaural audio pairs spanning diverse real-world scenes and camera rotation trajectories, constructed through a semi-automated pipeline. Furthermore, we propose ViSAudio, an end-to-end framework that employs conditional flow matching with a dual-branch audio generation architecture, where two dedicated branches model the audio latent flows. Integrated with a conditional spacetime module, it balances consistency between channels while preserving distinctive spatial characteristics, ensuring precise spatio-temporal alignment between audio and the input video. Comprehensive experiments demonstrate that ViSAudio outperforms existing state-of-the-art methods across both objective metrics and subjective evaluations, generating high-quality binaural audio with spatial immersion that adapts effectively to viewpoint changes, sound-source motion, and diverse acoustic environments. Project website: https://kszpxxzmc.github.io/ViSAudio-project.

1. Introduction

The introduction argues that video-to-binaural generation remains unexplored despite limits in mono-first spatialization, and presents BiAudio and ViSAudio as the dataset and framework for addressing this gap.

1. Introduction

  • Binaural audio can provide spatial awareness for immersive audio-visual experiences, but traditional production requires specialized equipment and expertise.The introduction motivates automatic generation from silent video as a practical alternative.Source: §1 · p. 2
  • Prior binaural methods commonly generate mono audio before spatialization, leaving performance dependent on the first-stage output and risking video misalignment and spatial inconsistency.Other end-to-end spatial-audio methods use 360° video or require camera-orientation parameters.Source: §1 · p. 2
  • BiAudio contains approximately 97,000 real-world video-binaural pairs with diverse camera rotation trajectories, addressing dataset scale, diversity, and fixed-viewpoint limitations.The dataset was constructed through a semi-automated pipeline.Source: §1 · p. 2
  • ViSAudio uses conditional flow matching, dedicated left- and right-channel branches, and a conditional spacetime module to balance channel consistency and spatial distinctiveness.The module extracts synchronization and spatial features from video and injects them into audio generation.Source: §1 · p. 2
  • The authors report that ViSAudio outperforms state-of-the-art methods in objective and subjective evaluations and adapts to viewpoint changes, sound-source motion, and diverse acoustic environments.The framework is described as producing immersive binaural audio aligned with the input video.Source: §1 · p. 2

2. Related Work

Related work traces video-to-audio generation from autoregressive and diffusion approaches to flow matching, while distinguishing two-stage spatialization from end-to-end spatial-audio generation.

2. Related Work

  • BiAudio is identified as the largest video-binaural dataset, with open-domain real-world sounds and varied camera trajectories.The comparison caption defines FoV as Field-of-View and FOA as First-order Ambisonics.Source: Table 1 · p. 3
  • Recent video-to-audio work adopts flow-matching models and multimodal conditioning, while ViSAudio adds spatial cues to generate binaural audio.The cited examples include Frieren and MMAudio.Source: §2 · pp. 2–3
  • Earlier visual spatial-audio methods generate mono audio first, then use visual cues or UNet-like models to produce binaural channels, with reported risks of visual misalignment and limited spatial consistency.Some methods localize and track visible sound sources before synthesizing spatial audio.Source: §2 · p. 3
  • End-to-end spatial-audio methods generate FOA from 360° video or FoV video with camera-direction conditioning, while end-to-end binaural generation remains unexplored.The paper positions ViSAudio as directly inferring binaural audio from visual content.Source: §2 · p. 3

3. BiAudio Dataset

BiAudio is a large-scale video-to-spatial-audio dataset built from real-world scenes, with captions and camera-motion diversity intended to preserve spatial cues across viewpoints.

3. BiAudio Dataset

  • BiAudio is a large-scale video-to-spatial-audio dataset comprising about 97,000 pairs of binaural spatial audio and perspective video clips, each lasting 8 seconds and accompanied by descriptive captions, totaling 215 hours.Source: §3 · p. 3

3.1. Dataset Construction

  • The construction pipeline projects 360° videos into 90° perspective views along camera trajectories and renders paired FOA audio into binaural signals using HRIR convolution.The source recordings come from the real-world, open-domain Sphere360 dataset.Source: §3.1 · p. 3
  • Two-stage captioning distinguishes visible sounds from off-screen sources and environmental noise to reduce semantic noise and protect audio-visual alignment.Qwen2.5-Omni creates detailed descriptions, which Qwen3-Instruct-2507 distills into concise visible- and invisible-sound captions.Source: §3.1 · p. 3
  • Camera trajectories vary initial pitch, yaw, and roll and add random drifts; clips with normalized left-right channel differences below 0.01 are discarded.Initial orientations are adjusted to keep the primary sound source roughly centered for part of the trajectory.Source: §3.1 · p. 4

3.2. Dataset Statistics

  • BiAudio is described as the largest FoV video dataset with binaural audio, spanning diverse real-world environments rather than the narrower domains of earlier datasets.The cited comparisons characterize OAP as street-focused and Fair-Play, SimBinaural, and MUSIC-21 as music-focused; SimBinaural also uses synthetic video and audio.Source: §3.2 · p. 4

4. ViSAudio Method

ViSAudio fine-tunes a pretrained audio generator and uses conditional flow matching to generate binaural audio from video and optional text. Its dual-branch architecture models channel-specific flows, while a conditional spacetime module integrates spatial and synchronization cues; the reported evaluation shows it outperforming all baselines across three test sets and all metrics.

4.1. Preliminaries

  • Conditional flow matching transforms noise into audio latents conditioned on video and text, with separate learned flows for the left and right channels.The resulting channel latents are decoded into mel spectrograms and then vocoded into a binaural waveform.Source: §4.1 · p. 4

4.2. Dual-Branch Audio Generation

  • Dual-branch generation uses correlated, synchronized left and right latents, combining shared multimodal alignment with independent channel-specific refinement.Video and text features capture coarse semantics, synchronization features capture event timing and dynamics, and spatial features provide channel-specific cues for localization.Source: §4.2 · pp. 4–5

4. ViSAudio Method

  • The architecture pairs independent left/right flow prediction with video-derived spatiotemporal conditioning to improve audio–video alignment.The pretrained MMAudio model is fine-tuned as the generation backbone.Source: §4 · p. 4, Figure 2 · p. 5

4.3. Conditional Spacetime Module

  • The Conditional Spacetime Module combines spatial features, synchronization cues, and global context, then injects the resulting features into branch transformer blocks.Channel-specific projections preserve distinctive binaural characteristics while representing spatial layouts and temporal dynamics.Source: §4.3 · pp. 5–6

4. ViSAudio Method

  • ViSAudio consistently outperforms all baselines across every reported metric on the BiAudio, MUSIC, and FAIR-Play test sets.ThinkSound and AudioX produce two-channel stereo rather than authentic binaural audio.Source: Table 2 · p. 6

5. Experiments

The experiments evaluate ViSAudio using objective and human-rated measures, qualitative comparisons, and component ablations. The reported results favor ViSAudio across benchmarks and perceptual criteria, while the ablation separates the spatial benefits of its components from a slight temporal-alignment trade-off.

5.1. Experimental Setup

  • BiAudio videos are segmented into 8-second clips at 44.1 kHz; splitting by video ID yields 94,845 training and 2,695 test clips, evaluated with distributional, synchrony, semantic metrics and five-aspect subjective MOS.Baselines include ThinkSound, AudioX, ViSAGe, and See2Sound; FAIR-Play is an out-of-distribution test set comprising 1,871 clips.Source: §5.1 · pp. 6–7

5.2. Quantitative Results

  • ViSAudio outperforms the baselines on overall quality, spatial channel matching, synchrony, and semantic alignment, with strong performance on unseen FAIR-Play data and across all subjective metrics.The user study reports state-of-the-art spatial impression, audiovisual consistency, and audio realism.Source: §5.2 · p. 7, Table 3 · p. 7

5.3. Qualitative Results

  • In the sitar example, ViSAudio’s binaural output most closely matches ground truth and reflects the source shifting as the camera moves left to right.Other methods show incorrect rhythms and fail to represent left-right channel differences.Source: §5.3 · pp. 7–8

5.4. Ablation Study

  • Adding the Conditional Spacetime Module improves spatial perception by 0.575 U-SC, while slightly decreasing temporal alignment.The Dual-Branch Audio Generation module improves spatial generation while retaining strong temporal and semantic alignment; the study evaluates FDavg VGG, DeSync, IB-Score, U-SI, and U-SC.Source: §5.4 · p. 8, Table 4 · p. 8

6. Conclusion

The conclusion presents ViSAudio and BiAudio as contributions to end-to-end video-driven binaural audio generation, while identifying short clips and recording noise as limitations. The appendix provides supplementary dataset, method, and experiment details.

6. Conclusion

  • ViSAudio combines dual-branch audio generation with a conditional spacetime module, and BiAudio supplies diverse real-world video-binaural pairs with camera motion.The conclusion reports state-of-the-art performance and adaptation to viewpoint changes.Source: §6 · p. 8
  • Short clips limit long-range temporal dependencies and complex acoustic events, while background noise in BiAudio recordings may introduce output artifacts.Future work aims to handle longer sequences and directly generate multichannel audio such as FOA.Source: §6 · p. 8

Appendix

  • The appendix covers dataset construction and statistics, ViSAudio details, experimental configurations, additional ablations, and demonstration cases.It also describes video demonstrations across acoustic scenarios.Source: Appendix · p. 9

A.1. Disclaimer on Copyright and Data Usage

The disclaimer describes the data sources and terms under which BiAudio’s video data are used, distinguishing Creative Commons material from other copyrighted videos.

A.1. Disclaimer on Copyright and Data Usage

  • BiAudio construction extends an existing dataset, and the authors state that they adhere to its data-use terms.The video data came from YouTube.Source: §A.1 · p. 9
  • Creative Commons videos are attributed to their original authors under CC BY 4.0 where applicable.Source: §A.1 · p. 9
  • For videos not under Creative Commons, the authors state that they are copyrighted and used only for academic research, with no intended commercial use.The disclaimer characterizes this use as falling under fair use for educational and research purposes.Source: §A.1 · p. 9

A.2. Dataset Construction Details

BiAudio construction aligns dynamic perspective video with spatially transformed FOA audio, while adding captions that distinguish visible from non-visible sounds and filtering for spatially distinct clips.

FOA representation

  • The FOA audio uses W as its omnidirectional component and X, Y, and Z for front-back, left-right, and up-down dimensions, respectively.These components describe the 3D sound field.Source: §A.2 · p. 9

Sound localization and viewpoint trajectory

  • Primary sound direction is estimated from spherical-harmonic energy, then camera trajectories begin near that direction and drift while keeping the main visual cues in view.Trajectories vary yaw, pitch, and roll to simulate natural camera motion.Source: §A.2 · pp. 9–10

Video and audio rendering

  • The pipeline rotates FOA audio segment by segment to match the camera trajectory, then renders the rotated ambisonics into binaural audio using Omnitone HRIRs.Perspective video is rendered with a fixed 90° field of view.Source: §A.2 · pp. 10–11

Caption generation

  • Qwen2.5-Omni descriptions of visible and background sounds, including off-screen sources, are refined by Qwen3-Instruct-2507 into structured captions.The pipeline is designed to distinguish visible from non-visible sound sources.Source: Figure R1 · p. 10, §A.2 · p. 11

Dataset filtering

  • Binaural clips are normalized channelwise and retained only when their mean absolute left-right difference exceeds 0.01.The threshold is used to keep clips with sufficiently strong inter-channel disparities.Source: §A.2 · pp. 11–12

A.3. Dataset Statistics Details

BiAudio statistics describe a broad sound vocabulary and the distribution and motion of camera viewpoints across clips.

Vocabulary

  • BiAudio captions contain 2,360 unique visible-sound terms and 1,265 unique invisible-sound terms.The authors interpret these vocabulary statistics as evidence of coverage across diverse acoustic environments.Source: §A.3 · p. 12

Initial Camera Viewpoint

  • The initial-viewpoint heatmap summarizes the distribution of yaw and pitch angles at the start of clips.It highlights which regions of the spherical field of view are most commonly observed.Source: §A.3 · p. 12

Camera Rotation Trajectory

  • A visualization of 500 sampled trajectories shows temporally evolving camera directions on a unit sphere, with color darkening over time.The curves show how cameras move across the field of view during each clip.Source: §A.3 · p. 12

B. GenDoP Method

The described method projects video spatial features into audio-aligned, channel-specific features. The accompanying experimental settings specify model training and the user-study procedure.

B.1. Frame-aligned Spatial Features

  • Learnable left- and right-channel spatial embeddings and convolutional and linear projections transform Perception Encoder features into frame-aligned features for audio generation.The resulting features are merged with synchronization features to provide spatio-temporal information to subsequent audio-generation modules.Source: §B.1 · p. 12

C.1. Experimental Settings

  • Experiments fine-tune MMAudio’s large 44.1 kHz v2 variant and train on BiAudio combined with MUSIC, which is oversampled three times.BiAudio inference removes the Visible and Invisible training labels, retaining sound descriptions for comparison across methods.Source: §C.1 · p. 13

C.2. User Study Details

  • The web-based user study has headphone-wearing participants rate five samples per video–audio group from 1 to 5 on five perceptual dimensions.The dimensions are Spatial Impression, Spatial Consistency, Temporal Alignment, Semantic Alignment, and Audio Realism.Source: §C.2 · p. 13

C.3. Additional Ablation Studies

The additional ablations report gains from combining model modules and training with BiAudio. Supplementary cases illustrate performance across dynamic sources, multiple sounds, off-screen events, and varied environments.

C.3. Additional Ablation Studies

  • The combination of the Dual-Branch Audio Generation and Conditional Spacetime modules significantly improves performance in the component ablation.Table S1 provides the complete objective-metric results supplementing Table 4 in the main text.Source: §C.3 · p. 13, Table S1 · p. 15

D. Additional Cases

  • The supplementary video presents additional binaural-audio cases in diverse acoustic environments alongside cases from the main text.It is intended to help viewers evaluate realism and spatial precision.Source: D. Additional Cases · pp. 13, 15

D.1. Dynamic Sound Sources

  • With fixed viewpoints, the model localizes static objects and tracks moving sources; with camera motion, it maintains stable spatial perception of stationary sources.The examples cover both object and viewpoint dynamics.Source: §D.1 · p. 15

D.2. Multiple Sound Sources

  • Examples cover identical sources at different positions, interacting sources, and overlapping ensembles, demonstrating spatial discrimination, separation, and localization.The authors describe the resulting auditory scenes as coherent, with accurate spatial perception.Source: §D.2 · p. 15

D.3. Invisible Sound Sources

  • When captions provide semantic guidance, ViSAudio generates off-screen sounds that are not visible in the input video.Examples show the model inferring and spatializing unseen events from contextual cues.Source: §D.3 · p. 15

D.4. Diverse Acoustic Environments

  • Examples in outdoor, underwater, and indoor scenes illustrate generalization across diverse acoustic environments.The authors report high-quality binaural audio with spatial immersion in these settings.Source: §D.4 · p. 15

D.5. Cases in the Main Text

  • The supplementary video also demonstrates the two Figure 1 cases and the Figure 3 case from the main text.Source: §D.5 · p. 15

Related papers

Loading 2512.03036v1…