PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformer

Ke Lei, Yu Zhang, Changhao Pan, Xueyi Pu, Wenxiang Guo, Ruiqi Li, Zhou Zhao

arXiv:2605.30940v1eess.AScs.MMcs.SD

TL;DR

Existing spatial-audio systems struggle to balance generation quality, inference speed, and accurate spatial alignment from multimodal inputs. SwanSphere addresses this with a streaming multimodal framework, achieving state-of-the-art performance across video-to-spatial-audio and text-to-spatial-audio benchmarks.

  • Problem

    Existing spatial-audio systems struggle to jointly maintain high quality, low latency, and accurate sound-source directionality for panoramic content.

  • Method

    SwanSphere combines a causal autoregressive diffusion transformer with SVAC, multi-objective ODPO, and automated MLLM-based spatial-caption annotation for multimodal streaming FOA generation.

  • Results

    Across video-to-spatial-audio and text-to-spatial-audio benchmarks, SwanSphere achieves state-of-the-art performance; its semantic-consistency FD is 120.28 versus OmniAudio’s 157.67.

  • Takeaways & Limitations

    SwanSphere supports high-fidelity, spatially precise multimodal audio synthesis with streaming inference and lower latency than baseline models.

  • Takeaways & Limitations

    The framework’s hyper-realistic spatial audio-visual synthesis could enable deceptive spatial deepfakes, motivating watermarking and non-commercial research-only licensing.

Abstract

from arXiv · show

Real-time and accurate spatial audio generation is pivotal for delivering an immersive experience. However, existing spatial audio synthesis technologies are often encumbered by a tradeoff between generation quality and high inference latency, as well as difficulty in capturing precise spatial information from multimodal inputs. To address these challenges, we propose SwanSphere, a unified streaming framework for high-fidelity spatial audio generation from panoramic videos and text prompts. SwanSphere mainly makes the following contributions: 1) We introduce a causal autoregressive diffusion transformer architecture that enables streaming high-quality spatial audio generation. 2) We design a Spatial Video-Audio Contrastive (SVAC) learning strategy to align the video encoder with the acoustic domain, and further employ a multi-objective online direct preference optimization (ODPO) scheme, resulting in strong spatial perception and robust multimodal spatial audio synthesis. 3) To alleviate the current scarcity of spatial audio datasets, we also develop an automated annotation pipeline for generating detailed spatial captions. Experimental results demonstrate that SwanSphere achieves superior performance in both video-to-spatial and text-to-spatial audio generation tasks. Demos can be found at: https://swanaigc.github.io.

1. Introduction

SwanSphere addresses the quality–speed and spatial-alignment challenges of spatial audio generation with a causal diffusion transformer for streaming multimodal synthesis. It combines multimodal spatial-awareness strategies, automated annotation, and comprehensive evaluations demonstrating strong audio quality, audio-visual alignment, and lower latency than baselines.

  • Motivation: Spatial audio generation must jointly maintain high quality and speed while accurately aligning audio with visual cues in omnidirectional videos.
  • SwanSphere: SwanSphere introduces a causal diffusion transformer architecture that enables streaming high-quality spatial audio generation from multimodal inputs.
  • Architecture: An autoregressive language model captures global patch-level temporal and spatial structures, while LocDiT performs local denoising and continuous spatial-audio rendering.
  • Data: An MLLM-based automated annotation pipeline creates spatial caption-FOA data from semantic, temporal, and spatial descriptions to support data scaling and model generalization.
  • Spatial alignment: SVAC and multi-objective ODPO strengthen multimodal spatial perception and align generated audio with human preferences across aesthetics, semantics, and spatial perception.
  • Evaluation: Comprehensive subjective and objective evaluations show strong performance in audio generation quality and audio-visual alignment, with lower latency than baseline models.

2. Related Work

Prior video-to-audio work has progressed from latent diffusion toward autoregressive and causal online generation, while spatial-audio methods still require explicit spatial guidance and often overlook video-based spatial cues. SwanSphere addresses these limitations with SVAC and Multi-Objective ODPO for improved spatial alignment.

  • Video-to-Audio Generation: V2A generation predominantly uses latent diffusion models, while emerging work investigates autoregressive token-based paradigms.This reflects recent advances in multimodal AIGC and video-to-audio synthesis.
  • Video-to-Audio Generation: V-AURA aligns high-frame-rate visual features with audio tokens temporally and models audio-video co-occurrence through cross-modal feature fusion.The approach improves temporal synchronization and semantic consistency.
  • Video-to-Audio Generation: SoundReactor enables frame-level online V2A through a causal decoder-only transformer with a diffusion head for low-latency stereo audio generation.Its design performs end-to-end causal modeling in an online setting.
  • Multimodal Spatial Audio Generation: Spatial-audio systems commonly synthesize mono audio before multimodal-guided spatialization, but prior methods mainly use semantic visual information and overlook explicit spatial cues.The cascaded paradigm is constrained by the quality of the initial audio generation.
  • Multimodal Spatial Audio Generation: SwanSphere proposes SVAC and Multi-Objective ODPO to refine spatial alignment and support more realistic spatial audio generation.The strategy responds to limitations in prior visual-semantic spatial-audio methods.

3. Method

SwanSphere generates streaming spatial audio by combining continuous FOA latent modeling, spatial audiovisual contrastive alignment, and a two-stage autoregressive diffusion architecture. Its training further uses multi-objective preference optimization and an automated corpus construction pipeline to improve spatial fidelity, multimodal alignment, and data scalability.

  • Audio Representation: FOA audio uses four channels for omnidirectional pressure and directional velocity components, while a fine-tuned Stable Audio VAE maps signals into continuous low-dimensional latents.The channels correspond to W, X, Y, and Z; the encoder produces z = E(a).
  • Spatial Video-Audio Contrastive Learning: SVAC aligns panoramic video and FOA audio across semantic, temporal, and spatial dimensions using VideoMAE features, temporal replication, and diverse hard negatives.Hard negatives include batch-wise semantic mismatches, temporally shifted audio, spatially rotated FOA, and horizontally rotated panoramic video.
  • Autoregressive Diffusion Generation: The autoregressive diffusion transformer divides synthesis into semantic planning and local generation, using a causal language model followed by a flow-matching Local Diffusion Transformer.Historical spatial audio and physically aware video features condition semantic embeddings, while the history encoder supports high-fidelity intra-patch generation.
  • Multi-Objective Preference Optimization: ODPO generates 8 candidates per input and ranks them using spatial, semantic, and acoustic-fidelity rewards to calibrate the model toward realistic spatial audio.ImageBind measures cross-modal semantic alignment, Audiobox Aesthetics measures perceptual fidelity, and the rewards are normalized to [0, 1].
  • Data Curation: An automated collection and annotation pipeline constructs the SwanSphere corpus from aligned panoramic video and spatial audio, while non-FOA audio supports curriculum learning.The pipeline addresses the scarcity of high-quality paired spatial-audio data and enables multimodal guidance and data scaling.

4. Experiments

SwanSphere is evaluated with objective and subjective measures of spatial audio fidelity and cross-modal alignment, outperforming baselines across video-to-spatial and text-to-spatial generation. Ablations further show benefits from SVAC, history conditioning, and larger model capacity.

  • Video-to-spatial audio: SwanSphere outperforms existing cascaded and end-to-end baselines across almost all objective metrics on the hybrid test set.Evaluation covers semantic quality, spatial precision, and efficiency, with objective measurements and subjective human studies.
  • Video-to-spatial audio: Total angular error 1.03 surpasses OmniAudio’s 1.27 and ViSAGe’s 1.59, while 1.09B parameters support 0.21-second first-chunk latency.The reported spatial accuracy is attributed to optimization in the ODPO stage.
  • Text-to-spatial audio: SwanSphere outperforms retrained OmniAudio for text-to-spatial audio, reducing FD from 174.13 to 142.8 and KL divergence from 1.83 to 1.43.It also leads the best-performing cascaded baseline, Tango2+AS, in semantic consistency and audio quality.
  • Ablation studies: Using generic frame-level CLIP features yields the poorest ablation results, with FD 140.28 and Angular Error 1.34, whereas SVAC enhances spatial accuracy and auditory fidelity.The result indicates that generic visual encoders lack sufficient domain adaptation for fine-grained spatial audio generation.
  • Ablation studies: Removing history conditioning increases FD from 120.28 to 128.15 and KL from 1.31 to 1.42, while smaller model variants consistently degrade performance.The capacity ablation compares SwanSphere-L (1.09B) with SwanSphere-M (0.62B) and SwanSphere-S (0.43B).

5. Conclusion

SwanSphere is a multimodal streaming framework for high-fidelity spatial audio generation from panoramic videos and text prompts. It separates semantic planning from local acoustic rendering while using contrastive learning and ODPO fine-tuning to enhance spatial representation.

  • SwanSphere generates high-fidelity spatial audio from panoramic videos and text prompts.
  • Its divide-and-conquer strategy decouples semantic planning from local acoustic rendering, supporting detailed reconstruction and low-latency streaming inference.
  • Spatial Video-Audio Contrastive Learning and multi-objective ODPO fine-tuning are introduced to enhance spatial representation.

Impact Statement · A. Dataset Construction Details

The framework synthesizes high-fidelity First-Order Ambisonics spatially aligned with panoramic video, supporting immersive applications while raising risks of hyper-realistic spatial deepfakes.

  • Impact Statement: The framework synthesizes high-fidelity First-Order Ambisonics spatially aligned with panoramic video content.The technology is presented as promising for VR/AR, the metaverse, and multimedia content creation.
  • Impact Statement: Its ability to generate spatially consistent audio-visual environments introduces risks of hyper-realistic spatial deepfakes.The passage warns that this capability could be misused to create deceptive misinformation.

A.1. Video-FOA dataset construction

The Video-FOA dataset combines Sphere360, YT-Ambigen, and newly collected YouTube video-FOA pairs to support diverse spatial audio generation. After re-clipping, deduplication, and integration, the consolidated dataset contains approximately 165,000 valid clips totaling 458 hours, with 5% reserved for testing.

  • Source datasets: Sphere360 contributes 103,000 360-degree video clips with FOA audio, totaling approximately 288 hours.Sphere360 is identified as one of the dataset’s component sources.
  • Source datasets: YT-Ambigen contributes 102,000 diverse spatial audio-visual clips, totaling approximately 142 hours.Its original 5-second clips were re-clipped to 10 seconds for framework consistency.
  • Source datasets: 50,000 newly collected YouTube video-FOA pairs totaling 138 hours address long-tail underrepresentation of rare audio events.The collection followed the data collection and cleaning pipeline in Liu et al. (2025b).
  • Consolidation and split: The final consolidated dataset contains approximately 165,000 valid clips totaling 458 hours, with 5% reserved as the test set.Redundant samples were removed because of significant video-content overlap between YT-Ambigen and Sphere360, after which the newly collected data were integrated.

A.2. Implementation details of the automated spatial captioning pipeline

The automated spatial captioning pipeline combines DSP algorithms with MLLMs to improve spatial azimuth descriptions through acoustic feature extraction, trajectory smoothing, and multimodal fusion. It estimates spatial trajectories from FOA audio, smooths them for physical consistency, and integrates them with panoramic video and downmixed audio.

  • Pipeline overview: The pipeline has three stages: acoustic spatial feature extraction, spatial trajectory smoothing, and multimodal fusion generation.It combines classical Digital Signal Processing (DSP) algorithms with advanced Multimodal Large Language Models (MLLMs).
  • Acoustic spatial feature extraction: DoA estimation applies STFT to four-channel FOA audio over 500Hz - 8000Hz, then derives azimuth and elevation from energy-weighted acoustic intensity vectors.The frequency band targets most human-speech and primary-environmental-sound energy, while energy weighting mitigates background-noise interference.
  • Spatial trajectory smoothing: Spatial smoothing converts angular data into three-dimensional unit vectors and uses a moving average with a window size of 3 to reduce fluctuations and computational noise.The unit-vector representation addresses the discontinuity at ±180◦, while the method also estimates relative proximity.
  • Multimodal fusion: Gemini 2.5 Pro receives structured spatial trajectories with the original panoramic video and downmixed audio to generate concise spatial audio descriptions.The descriptions include sound-source content, azimuth or location, approximate timing, and dynamics such as the Doppler effect.

B. Implementation Details · B.1. Latent Representation of Spatial Audio via 4-channel FOA-VAE

The implementation uses a 4-channel FOA-VAE initialized from Stable Audio VAE weights and adapted for spatial audio representation. Training uses staged mixed-precision optimization with specified hardware, step counts, learning rates, and reconstruction objectives.

  • B. Implementation Details: B. Implementation Details: The spatial VAE is initialized with pre-trained Stable Audio VAE weights to leverage existing non-spatial audio knowledge.
  • B.1. Latent Representation of Spatial Audio via 4-channel FOA-VAE: B.1. Latent Representation of Spatial Audio via 4-channel FOA-VAE: The standard framework removes the MS-STFT used for stereo reconstruction.
  • B.1. Latent Representation of Spatial Audio via 4-channel FOA-VAE: B.1. Latent Representation of Spatial Audio via 4-channel FOA-VAE: The modified system transforms left/right components into FOA components W, X, Y, and Z.
  • B.1. Latent Representation of Spatial Audio via 4-channel FOA-VAE: B.1. Latent Representation of Spatial Audio via 4-channel FOA-VAE: Mixed-precision training uses batch size 80 across 2 NVIDIA H800 GPUs for 200,000 steps.
  • B.1. Latent Representation of Spatial Audio via 4-channel FOA-VAE: B.1. Latent Representation of Spatial Audio via 4-channel FOA-VAE: The VAE encoder is frozen before decoder training continues for an additional 300,000 steps.
  • B.1. Latent Representation of Spatial Audio via 4-channel FOA-VAE: B.1. Latent Representation of Spatial Audio via 4-channel FOA-VAE: AdamW uses a generator learning rate of 1 × 10−5 and a discriminator learning rate of 2 × 10−5.
  • B.1. Latent Representation of Spatial Audio via 4-channel FOA-VAE: B.1. Latent Representation of Spatial Audio via 4-channel FOA-VAE: The joint objective includes weighted four-channel multi-resolution STFT loss and KL divergence loss at the VAE bottleneck.
  • B.1. Latent Representation of Spatial Audio via 4-channel FOA-VAE: B.1. Latent Representation of Spatial Audio via 4-channel FOA-VAE: The L1 loss weight is zero because it significantly interfered with high-frequency reconstruction.

B.2. Training Details

SwanSphere training freezes pretrained VideoMAE-V2 and AudioMAE encoders while optimizing projection layers for shared video-audio alignment, then trains the model with AdamW and online DPO fine-tuning on a curated panoramic video-FOA dataset.

  • SVAC training: SVAC initializes VideoMAE-V2 and AudioMAE encoders, freezes both, and optimizes only 6.13M audio and 6.82M video projection-layer parameters.The projection layers map semantic audio and video features into a shared video-audio alignment space.
  • SwanSphere training: SwanSphere uses AdamW at a learning rate of 1 × 10−5 for 600,000 steps on 8 NVIDIA H800 GPUs and 458 hours of curated mixed panoramic video-FOA data.These settings define the main training configuration.
  • DPO training: The subsequent multi-objective DPO stage performs three rounds of online fine-tuning to align generative outputs with spatial and semantic preferences.Online fine-tuning follows the main SwanSphere training stage.

B.3. Data for curriculum learning

SwanSphere uses an approximately 1M-sample pre-training dataset of non-spatial audio adapted into pseudo-FOA format to improve generalization and support spatial generation.

  • B.3. Data for curriculum learning: Approximately 1M non-spatial audio samples from AudioCaps, VGGSound, WavText5k, and AudioSet form SwanSphere’s pre-training dataset.The dataset is constructed to enhance SwanSphere’s generalization.
  • B.3. Data for curriculum learning: The non-spatial signals are adapted into pseudo-FOA format for use within the spatial generation framework.This adaptation enables non-spatial audio to contribute to spatial audio generation.

C. More experiments

SwanSphere is evaluated with an independent spatial metric and under out-of-distribution conditions. It achieves the best reported performance on unseen environments and microphone rigs across semantic and spatial measures.

  • SELD spatial evaluator: The study uses weighted cosine similarity (wCS) from a pretrained SELD model as an independent spatial evaluator.This avoids coupling the spatial reward with the evaluation protocol.
  • Out of distribution evaluation: Out-of-distribution evaluation uses YT360-Test videos from unseen environments and microphone rigs.The evaluation compares SwanSphere with cascaded and end-to-end baselines.
  • Out of distribution evaluation: SwanSphere achieves the best performance on YT360-Test for semantic quality and spatial accuracy, demonstrating robustness to out-of-distribution conditions.Semantic quality is measured with FD and KL, while spatial accuracy uses angular metrics.
Loading 2605.30940v1…