OmniAudio: Generating Spatial Audio from 360-Degree Video

1. Introduction

• 360V2SA generates FOA spatial audio directly from 360-degree video, preserving comprehensive visual context for sound-emitting objects and their spatial relationships.The task addresses non-spatial audio and restricted perspective-video inputs without requiring camera-angle control.Source: §1 · p. 1
• The task faces scarce paired data, full-sphere audio-visual synchronization, and high-fidelity spatial-audio generation challenges.Source: §1 · p. 1
• OmniAudio combines coarse-to-fine self-supervised pre-training, latent flow matching, and dual-branch panoramic-perspective video processing for spatial-audio generation.The branches capture local object details and global spherical context, while pre-training transfers patterns from large-scale non-spatial audio data.Source: §1 · p. 2, §2 · p. 3
• The work establishes Sphere360-Bench as a standardized evaluation framework where OmniAudio achieves state-of-the-art performance across objective and subjective metrics.Source: §1 · p. 2

3. The Sphere360 Dataset

• Sphere360 contains over 103,000 paired 360-degree video and audio clips totaling 288 hours, covering diverse real-world acoustic environments and noise conditions.The dataset is partitioned by video IDs to preserve evaluation integrity and prevent leakage.Source: §1 · p. 2, §3 · p. 3
• Sphere360 is presented as the largest 360-degree video dataset with FOA audio, with stricter collection and cleaning protocols than YT-360.The pipeline filters stationary, silent, speech-heavy, and audio-visually mismatched content using semi-automated procedures.Source: §3 · pp. 3–4

4. OmniAudio

• OmniAudio uses coarse-to-fine self-supervised flow-matching pre-training followed by efficient video-guided fine-tuning for spatial-audio generation.The two stages use unlabeled spatial and non-spatial audio, then integrate panoramic video representations during fine-tuning.Source: §4 · p. 4, §4.4 · p. 5, §4.5 · p. 6

4.2. Spatial Audio Representation

• The four-channel VAE adapts a pre-trained stereo VAE to encode FOA channels W, X, Y, and Z for spatial audio.The modification removes stereo-specific MS-STFT processing and assigns each FOA component a loss weight of 1/4.Source: §4.2 · p. 4

4.3. Dual-Branch Video Representation

• The dual-branch video encoder combines global equirectangular panoramic features with local perspective features to model scene context and fine-grained field-of-view details.Both branches use a frozen MetaCLIP-Huge encoder, and their representations are fused through diffusion transformers.Source: §4.3 · p. 5

4.4. Self-supervised Flow Matching Pre-training

• The first pre-training phase converts abundant non-spatial audio into FOA, compresses it with the spatial VAE, and applies token masking to learn general audio structure.The velocity field is conditioned on partially masked audio latents, with random masked frames and a minimum masking span.Source: §4.4 · p. 5
• The second pre-training phase uses only FOA audio to align representations with spatial dynamics and improve high-fidelity FOA generation.Its objective is applied only to masked pieces.Source: §4.4 · pp. 5–6

4.5. Spatial-Aware Supervised Fine-tuning

• During fine-tuning, upsampled perspective features are added to the audio latent sequence, while pooled panoramic features provide global conditioning to the diffusion transformer.Inference follows the learned velocity field and then decodes the result through the VAE into FOA audio.Source: §4.5 · p. 6

5.2. Main Results

• OmniAudio achieves FD 92.57 and KL divergence 1.64 on YT360, outperforming Diff-Foley + AS and MMAudio + AS on non-spatial metrics.On Sphere360, it achieves FD 88.30 and KL divergence 1.58, also surpassing both cascaded baselines.Source: §5.2 · pp. 6–7, Table 2 · p. 7
• OmniAudio attains ∆Angular 1.27 on YT360 and 1.28 on Sphere360, lower than the reported ViSAGe baselines.The comparisons are 1.99 and 1.98 for ViSAGe on YT360, and 1.49 and 1.51 on Sphere360.Source: §5.2 · p. 7, Table 2 · p. 7
• Human evaluators give OmniAudio MOS-SQ 84.67 and MOS-AF 87.23, the highest reported perceptual-quality and video-audio-alignment scores.The passage notes relatively large MOS standard deviations because spatial-audio assessment is subjective.Source: §5.2 · p. 7, Table 2 · p. 7

5.3. Case Study

• Qualitative comparisons show OmniAudio generating FOA audio for an object moving behind the camera and preserving musical rhythm more consistently than baselines.The observations associate full spherical coverage with reconstructing out-of-view sound sources and higher spectrogram fidelity and alignment.Source: §5.3 · p. 7, Figure 3 · p. 8

5.4. Ablation Study

• Coarse-to-fine self-supervised pre-training outperforms spatial-only, non-spatial-only, and no-pre-training configurations across all metrics.Removing pre-training or the coarse stage causes notable degradation, supporting the use of both spatial and non-spatial audio data.Source: §5.4 · pp. 7–8, Table 3 · p. 8
• The dual-branch design outperforms single-branch variants across metrics, while larger models perform better and the Large 1.2B configuration is strongest.Panoramic inputs outperform perspective inputs, ERP plus perspective features capture complementary information, and reducing model size substantially degrades fidelity and spatial precision.Source: §5.4 · pp. 8–9, Table 4 · p. 8, Table 5 · p. 9

• OmniAudio achieves state-of-the-art performance across objective and subjective metrics on Sphere360.Source: §6 · p. 9
• The paper proposes 360V2SA and uses FOA audio to represent spatial cues from 360-degree video.Source: §6 · p. 9
• Sphere360 is a large-scale dataset curated for 360V2SA, supported by a semi-automated collection and cleaning pipeline.Source: §6 · p. 9
• OmniAudio combines self-supervised pre-training with a dual-branch architecture to capture local and global information from panoramic video.Source: §6 · p. 9

6.1. Positive Impacts

• Generated audio could improve accessibility, content creation, education, immersion, and historical preservation.Source: §6.1 · p. 9

6.2. Negative Impacts

• Realistic audio generation from video could facilitate misinformation or deepfakes that misrepresent people’s words or actions.Source: §6.2 · p. 9
• Using personal videos without consent could create privacy violations.Source: §6.2 · p. 9
• Increasingly capable audio generation may displace professionals in audio editing and sound design.Source: §6.2 · p. 9
• Generated audio resembling copyrighted music or sound effects could create legal complications.Source: §6.2 · p. 9

6.3. Key Considerations

• The statement calls for guidelines, regulation, transparency, and responsible development to mitigate potential harms.Source: §6.3 · p. 9

4. Model Configurations and Architecture

• The paper includes figures for model configurations and a variational autoencoder.Source: §4 · p. 13
• The architecture section also includes a diffusion-transformer figure.Source: §4 · p. 13

5. Evaluation

• The evaluation section includes figures for objective metrics, subjective metrics, spatial audio format, and audio spatialization.Source: §5 · p. 13

6. Limitations and Future Work

• The supplied figure labels identify limitations and future work, plus additional results for classifier-free-guidance scale and spatial VAE effects.Source: §6 · p. 13, §7 · p. 13

A. Disclaimer on Copyright and Data Usage

• Non-Creative-Commons videos are used exclusively for academic research, while Creative Commons videos receive attribution where applicable.Source: A. Disclaimer on Copyright and Data… · p. 13

B.1. Data Crawling

• The crawling pipeline searches for 360-degree and FOA content using event-label keywords, technical filters, and channel- and video-based collection stages.Source: §B.1 · pp. 14–15
• The resulting collection contains 166.5K clips from 7179 videos, totaling approximately 462.5 hours.Source: §B.1 · p. 15

B.2. Data Cleaning

• Cleaning removes stationary videos, silent audio, excessive speech, and audio-visual mismatches using thresholded automated checks.Source: §B.2 · pp. 15–16
• Manual inspection samples removed and retained data to validate cleaning decisions and adjust thresholds when discrepancies are found.Source: §B.2 · p. 16

B.3. Pipeline Statistics

• Approximately 7,500 silent, 12,000 static, 34,000 excessive-human-voice, and 23,500 mismatched clips were removed during cleaning.Some removed clips overlapped across categories, and the cleaned-category counts are shown in Figure 7.Source: §B.3 · p. 17, Figure 7 · p. 17

C. Dataset Comparisons and Benchmark Construction

• Sphere360 contains over 103,000 high-quality 360-degree clips with FOA audio totaling 288 hours, while Sphere360-Bench selects around 3,000 clips covering about 180 audio events.Benchmark construction combines automated screening, class filtering, confidence-based selection, and iterative manual verification.Source: Table 8 · p. 17, C. Dataset Comparisons and Benchmark… · p. 18, Figure 8 · p. 18

D. Model Configurations and Architecture

• The spatial VAE reconstructs FOA audio using weighted four-channel multi-resolution STFT, KL-divergence, and discrimination losses.It is initialized from a stereo VAE, with the architecture illustrated in Figure 9.Source: D. Model Configurations and Architecture · pp. 18, 20, Figure 9 · p. 19
• OmniAudio’s flow component uses a 24-layer, 24-head Diffusion Transformer with 1,536-dimensional embeddings and 1.2 billion parameters.The transformer uses local and global conditioning dimensions of 768 and 1,536, respectively.Source: Figure 10 · p. 19, D. Model Configurations and Architecture · p. 20, Table 9 · p. 20

E.1. Objective Metrics

• Fréchet Distance compares generated and reference audio feature distributions in OpenL3 space, with lower values indicating closer distributions.KL divergence additionally measures differences between generated and reference label distributions for semantic alignment.Source: §E.1 · pp. 20–21
• Spatial accuracy is evaluated from FOA directional-channel intensities Ix = mean(W · X), Iy = mean(W · Y), and Iz = mean(W · Z).These intensities support metrics for azimuth, elevation, and overall direction-of-arrival errors.Source: §E.1 · pp. 20–21
• Theta Error ∆absθ measures azimuth differences using a circular difference between ground-truth and estimated angles.Azimuth is the angle in the horizontal plane.Source: §E.1 · p. 20
• Phi Error ∆absϕ measures the absolute difference between ground-truth and estimated elevation angles.Elevation is treated as the vertical angular coordinate.Source: §E.1 · p. 20
• Spatial-Angle Error ∆absSpatial-Angle measures the difference between ground-truth and estimated directions of arrival.Its computation combines the azimuth and elevation differences.Source: §E.1 · p. 21

E.2. Subjective Metrics

• MOS raters score audio naturalness on a 20-100 Likert scale while focusing on quality, naturalness, spatiality, and overall preference.The test probes perceived spatial-audio quality.Source: §E.2 · p. 21
• Video-audio alignment raters judge whether spatial audio faithfully aligns with 360-degree video using completely, mostly, or somewhat responses on a 20-100 scale.This test targets correspondence between the generated spatial audio and visual content.Source: §E.2 · p. 21
• The study recruits 15 in-person participants, each rating 50 randomly selected Sphere360-Bench samples.Participants were used instead of crowdsourcing because of the characteristics of 360-degree video and spatial audio.Source: §E.2 · p. 21

E.3. Spatial Audio Format

• FOA encodes spatial audio with W, X, Y, and Z channels representing overall pressure, front-back, left-right, and vertical information.This balances spatial resolution with computational simplicity.Source: §E.3 · p. 21

E.4. Audio Spatialization Component

• Mono audio is spatialized into FOA using W = 1/√2s(t), X = cos θ cos ϕs(t), Y = sin θ cos ϕs(t), and Z = sin ϕs(t).The generated mono signal is encoded at the corresponding direction in the ambisonic domain.Source: §E.4 · p. 21

F. Limitations and Future Work

• OmniAudio struggles with samples containing many sound-emitting objects, including cases where instrument sounds are misclassified as applause.The limitation arises in complex real-world scenes with multiple simultaneous targets.Source: F. Limitations and Future Work · p. 21, Figure 11 · p. 22
• The cleaned dataset contains 100,000 samples, which the authors describe as insufficient for robust real-world 360V2SA and plan to expand through continued pipeline-based collection.Future work also targets better understanding of 360-degree videos with multiple targets.Source: F. Limitations and Future Work · p. 22

G. Additional Quantitative Results

• CFG-Scale experiments compare changes in FD and ∆abs Spatial-Angle, leading the authors to choose CFG-Scale = 5 to balance them.Figure 12 presents the metric variation across settings.Source: G. Additional Quantitative Results · p. 22, Figure 12 · p. 22
• The proposed spatial VAE outperforms descript-audio-codec and a non-spatial VAE across STFT, MEL, FD, and KL metrics, where lower values are better.The comparison is reported in Table 10.Source: G. Additional Quantitative Results · p. 22, Table 10 · p. 22
• The original ERP+Front dual-branch strategy achieves the best FD and KL while maintaining similar spatial metrics to multiple-FOV alternatives.The comparison covers hexadirectional, quadrant, and bipolar cuts, supporting the final single-frontal-view design.Source: G. Additional Quantitative Results · p. 23

• The train example shows the model preserving the train sound after it leaves the frontal view, while other models almost entirely fade.The rectangular annotation marks the continued audio capture beyond the frontal perspective.Source: Figure 13 · p. 24
• Fireworks rising and exploding, alongside motorcycles chasing on a dirt road, provide scenarios with intense wind and engine sounds.Source: Figure 14 · p. 25
• Boat navigation through waves and movement through a noisy indoor crowd provide additional water-splashing and crowd-noise scenarios.Source: Figure 15 · p. 25