PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldSonus: Bringing Sound to Worlds

Pengjun Fang, Jingyi Fa, Kam Man Wu, Jiaming Wang, Haoyuan Huang, Yaguang Wu, Xiangjun Huang, Ziyang Ma, Weijia Chen, Hongyu Liu, Zeyue Tian, Qifeng Chen

arXiv:2610.08760v1cs.SDcs.AIcs.CVeess.AS

TL;DR

Interactive world models create evolving visuals but remain largely silent, and existing approaches do not jointly provide modular causal streaming, mid-stream control, and spatially aligned stereo. WorldSonus combines chunk-wise causal audio generation, dynamic prompts, and stereo supervision; it matches or exceeds offline bidirectional baselines across open-domain and interactive videos while generating at RTF = 0.41.

  • Problem

    Interactive world-model sound must track evolving visuals, camera viewpoint, and updated instructions, while existing approaches lack a modular framework combining causal real-time generation, mid-stream control, and perspective stereo.

    Source: §1 · p. 1

  • Method

    WorldSonus uses streaming causal AR-diffusion with bounded state, mid-stream prompt scheduling, stereo supervision, and training-only ShiftNCE for temporal alignment.

    Source: §1 · p. 2, §3.5 · p. 5

  • Results

    WorldSonus matches or exceeds offline bidirectional V2A baselines across VGGSound and Interactive videos, with RTF = 0.41.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    WorldSonus demonstrates that causal streaming audio can achieve competitive or superior acoustic quality and spatial localization across open-domain videos and interactive environments.

    Source: §1 · p. 2, §6 · p. 9

    Authentic high-quality stereo data remains scarce, and expanding its scale remains a future goal.

    Source: §7 · p. 10

Abstract

from arXiv · show

Recent advances in world models have enabled increasingly realistic visual synthesis. However, these generated environments remain largely silent. Bringing sound to world models poses three core challenges: real-time generation to keep pace with interactive video streams, interactive control to respond to mid-stream sound instructions, and spatially aligned stereo to reflect scene geometry and camera motion. To address these demands, we introduce WorldSonus, an interactive video-to-audio framework designed for real-time spatial sound synthesis in world models. For real-time generation, WorldSonus employs a streaming causal autoregressive diffusion architecture that synthesizes audio chunks at a low real-time factor (RTF) of 0.41. For interactive control, we incorporate an audio-centric captioning pipeline with chunk-indexed prompt scheduling, enabling dynamic manipulation of sound events during generation. For spatial alignment, we leverage high-quality stereo supervision curated from diverse stereo and ambisonic data. Extensive experiments demonstrate that while tailored for world models, WorldSonus generalizes effectively to open-domain video-to-audio benchmarks, matching or outperforming state-of-the-art bidirectional models in both acoustic quality and spatial alignment. Project page: https://noizai.github.io/WorldSonus/

1 INTRODUCTION

Interactive world models need sound that tracks evolving visuals, viewpoint, and changing instructions, but existing approaches do not provide all three capabilities in a modular streaming system. WorldSonus combines causal real-time generation, mid-stream prompt control, stereo alignment, and training-only synchronization supervision.

Motivation

  • World models synthesize interactive visual environments, but adding sound requires tracking evolving visuals, camera-aligned spatial balance, and updated text instructions.Existing approaches either couple audio to a visual backbone or lack one or more of modularity, causal streaming, interactive control, and perspective stereo.Source: §1 · p. 1

Approach

  • WorldSonus generates 100 ms audio chunks from past and current visual context, updates prompts mid-stream without resetting, and uses clear-stereo and view-aligned FOA supervision.A bounded persistent state keeps memory and compute constant as the stream expands.Source: §1 · p. 2

Contributions

  • Two-timescale visual conditioning pairs chunk-level summaries with frame-aligned local cues, while training-only ShiftNCE refines event timing without inference cost.The main contributions unify causal real-time generation, mid-stream prompt responsiveness, camera-aligned stereo, and synchronization training without runtime latency.Source: §1 · p. 2

Experiments

  • WorldSonus matches or exceeds state-of-the-art offline bidirectional V2A baselines across VGGSound and Interactive videos, with VGGish FAD scores of 1.73 and 2.03, respectively, and RTF = 0.41.Latency is 41.2 ms per 100 ms audio chunk on one NVIDIA H100 GPU.Source: §1 · p. 2

2 RELATED WORK

Prior work advances bidirectional video-to-audio generation, streaming synthesis, and interactive visual worlds, but does not combine causal streaming audio with dynamic control in a persistent session. WorldSonus instead treats audio as a separate causal module conditioned on an external visual stream.

Bidirectional video-to-audio generation

  • Bidirectional V2A and spatial-audio approaches improve semantic, temporal, and spatial generation, but require full-clip visual context and lack streaming input or dynamic control.The described approaches include text-conditioned, reference-audio-conditioned, and multi-modality generation.Source: §2 · p. 2

Streaming video-to-audio generation

  • V-AURA, SwanSphere, and SoundReactor explore streaming, but their described systems do not establish fully end-to-end causal streaming; WorldSonus targets bounded state and mid-stream instruction changes.V-AURA uses bidirectional visual windows and non-causal decoding; SwanSphere lacks stateful stream decoding.Source: §2 · pp. 2–3

Interactive world models and joint audio–video generation

  • Unlike joint audio-video generation, WorldSonus conditions a separate audio module on realized external frames, without requiring access to the upstream model’s action space or internal architecture.Users or agents can update audio prompts independently.Source: §2 · p. 3

3 WorldSonus

WorldSonus generates stereo audio chunk by chunk from streaming video and changing prompts, using bounded recurrent memory, multi-scale visual conditioning, and contrastive temporal alignment.

3.1 STREAMING FORMULATION

  • The streaming formulation predicts each audio latent chunk from current video frames, the active prompt, and bounded history, using no future frames.Each 100 ms chunk corresponds to three latent frames aligned with three video frames; a stateful decoder emits 48 kHz stereo without re-decoding past latents.Source: §3.1 · p. 3

3.2 AUTOREGRESSIVE DIFFUSION

  • A fixed-capacity Ring-KV cache preserves session state with bounded memory, while the AR hidden state conditions a rectified-flow head that denoises each chunk’s latent frames.The flow head attends within the current chunk and uses frame-aligned visual tokens, without access to future blocks.Source: §3.2 · p. 3

3.3 TWO-TIMESCALE VISUAL CONDITIONING

  • Two-timescale visual conditioning routes a chunk summary to the AR backbone for semantic continuity and frame-aligned tokens directly to the flow head for local spatial and temporal detail.Frame tokens are derived from spatial patch grids and temporal differences; the figure depicts these two conditioning routes.Source: §3.3 · pp. 3–4, Figure 1 · p. 4

3.4 INTERACTIVE PROMPT CONTROL

  • At a chunk boundary, prompt updates replace the cached text cross-attention tokens while preserving the Ring-KV cache and existing visual and audio states.Training includes time-varying prompt schedules with instruction switches and drops, enabling steering within a persistent causal session.Source: §3.4 · p. 4

3.5 TRAINING

  • Teacher-forced pretraining applies rectified flow to stereo video-audio and audio-only data, with teacher forcing over bounded context; training also selects the best-matching of three candidate source noises per step.The latter uses Explorative Modeling during flow-matching training.Source: §3.5 · pp. 4–5
  • A frozen synchronization teacher supplies temporal targets for training-only ShiftNCE, which contrasts aligned and time-shifted embeddings and filters visually similar shifted candidates.The final pretraining loss combines flow and synchronization losses, followed by fine-tuning on a high-quality stereo video-audio subset.Source: §3.5 · p. 5

4 DATA CURATION

WorldSonus combines paired stereo video-audio, audio-only recordings, and panoramic data, with filtering and audio-centric captions supporting alignment and spatial supervision.

Data sources

  • 1,465 hours of audio comprise 999 hours of paired stereo video-audio and 466 hours of audio-only data from multiple public datasets and panoramic recordings.Panoramic clips are used to sample perspective crops and render view-aligned stereo audio.Source: §4 · p. 5

Data filtering

  • A two-stage pipeline removes technically poor stereo samples, then filters severe audiovisual mismatches and non-diegetic audio such as voiceovers and background music.Automated processing removes silent, phase-corrupted, narrow-stereo, and dual-mono samples before Qwen3-Omni verification.Source: §4 · p. 5

Captioning

  • Dense audio-centric captions describe sound categories, texture, timing, and spatial ambiance while excluding unrelated visual details.Video-audio captions use both modalities, whereas audio-only captions are generated from audio alone.Source: §4 · p. 5

5 EXPERIMENTS

Across the evaluated splits, WorldSonus combines causal streaming with competitive acoustic quality, stereo alignment, and interactive prompt control. The experiments also examine long-horizon stability and the effects of its alignment and generation design choices.

5.1 EXPERIMENTAL SETUP

  • Evaluation covers clear-stereo VGGSound and interactive gameplay and real-world footage, with acoustic, audiovisual-alignment, and stereo-balance metrics.The test splits include 5 s, 10 s, and 30 s clips; Greatest Hits collision events are also evaluated.Source: §5.1 · pp. 5–6, §5.2 · p. 6

5.3 MAIN RESULTS

  • Across five held-out splits, WorldSonus delivers competitive or superior acoustic quality to bidirectional models while generating 100 ms chunks in 41.2 ms (RTF = 0.41).The comparison covers five splits; the reported execution latency is on a single NVIDIA H100 GPU.Source: Table 1 · p. 6, §5.3 · pp. 6–7
  • WorldSonus achieves the highest BiasSkill across evaluated splits and reproduces left/right source dominance; bidirectional models have lower DeSync, while its event timing remains competitive without test-time guidance.Figure 2 provides qualitative evidence for two interactive clips; Greatest Hits is the model-free onset benchmark.Source: §5.3 · p. 7, Table 2 · p. 7, Table 3 · p. 7
  • On Interactive 30 s, rollout-tail FAD 2.51 versus direct-generation FAD 2.63 and DeSync 0.827 versus 0.880 indicate close tracking under a bounded 5 s Ring-KV cache.The comparison uses the final 25–30 s window from 1,024 clips.Source: §5.3 · p. 7, Table 4 · p. 8
  • WorldSonus reaches dual prompt-match rates of 25.68% on VGGSound and 23.44% on Interactive 10 s, and paired prompt switches yield positive intervention gains.The gains are +0.0509 and +0.0525, with 95% bootstrap confidence intervals excluding zero.Source: §5.3 · p. 8, Table 5 · p. 8, Table 6 · p. 8

5.4 ABLATION STUDIES

  • Removing ShiftNCE worsens DeSync from 0.831 to 1.067; direct 640 ms Synchformer feature injection also yields worse timing and acoustic quality.The authors suggest the 640 ms feature window may mismatch the 100 ms streaming chunks; ShiftNCE adds no inference overhead.Source: §5.4 · p. 9
  • DINOv3 patch features raise ImageBind alignment from 24.56 to 28.05 over pooled SigLIP 2, while removing the direct frame path raises FAD from 2.42 to 15.82.Removing temporal deltas worsens DeSync (0.958) and FAD (3.08).Source: §5.4 · p. 9
  • A 33 ms chunk size worsens acoustic quality (FAD 3.79), while 200 ms increases latency without consistent metric improvements; 100 ms is chosen as a practical balance.The comparison tests 33 ms, 100 ms, and 200 ms chunks.Source: §5.4 · p. 9, Table 8 · p. 9

6 CONCLUSION

WorldSonus is a causal spatial-audio framework for real-time sound generation in interactive world models. It combines chunk-wise flow matching, bounded visual context, and training-only synchronization to deliver competitive audio quality and spatial localization while supporting prompt updates.

6 CONCLUSION

  • WorldSonus generates 48 kHz stereo audio in 100 ms chunks, with 41.2 ms execution latency per chunk (RTF = 0.41) on a single NVIDIA H100 GPU.Source: §6 · p. 9
  • A bounded 5 s Ring-KV cache supports long-term planning while frame-aligned local guidance conditions the flow head.Source: §6 · p. 9
  • Training-only ShiftNCE enforces temporal synchronization without inference overhead, and the system supports real-time prompt updates.Source: §6 · p. 9
  • Across open-domain videos and interactive environments, WorldSonus achieves acoustic fidelity and spatial localization competitive with or exceeding offline bidirectional models.Source: §6 · p. 9

7 LIMITATIONS

The authors identify limitations in both causal audio representations and stereo training data. They point to more expressive causal codecs and expanded authentic stereo data as future needs.

7 LIMITATIONS

  • The causal VAE decoder has lower reconstruction quality than its non-causal counterpart, motivating more expressive, large-scale causal audio codecs.Source: §7 · p. 10
  • High-quality spatial audio data is scarce, and many public stereo videos have artificial or noisy channel separation that does not accurately reflect visual object motion.Source: §7 · p. 10
  • The authors identify expanding the scale of authentic stereo data as a key future goal; their filtering and panoramic soundfield decoding provide reliable spatial supervision.Source: §7 · p. 10

A IMPLEMENTATION CONFIGURATION AND ARCHITECTURAL DETAILS

This appendix summarizes WorldSonus's implementation configuration and architectural choices. It describes bounded-context streaming, independent visual and text guidance, and baseline evaluation procedures.

A IMPLEMENTATION CONFIGURATION AND ARCHITECTURAL DETAILS

B TRAINING RECIPE AND OPTIMIZATION DETAILS

The appendix details training, synchronization, prompt scheduling, corpus composition, and stereo-data curation. It describes the data filtering and captioning pipeline alongside the training procedures.

B TRAINING RECIPE AND OPTIMIZATION DETAILS

B.1 EXPLORATIVE MODELING FOR FLOW MATCHING (XM3)

  • During pretraining, XM3 samples three noises at the same timestep and dropout mask, computes their vector-field errors without gradients, and backpropagates only through the lowest-error sample.Source: §B.1 · p. 18

B.2 AMBIGUITY-AWARE CAUSAL SHIFTNCE

  • ShiftNCE distills synchronization signals from a frozen Synchformer teacher into the AR backbone during training and is disabled at inference; negative mining filters highly similar temporal offsets and selects the top 40% most teacher-separable anchors.Source: §B.2 · p. 18

B.3 DYNAMIC PROMPT SCHEDULES

  • The 10-second prompt schedules include a mid-stream switch at five seconds, withdrawal or arrival of text instructions, and sustained prompting across the sequence.Source: §B.3 · p. 19

C.1 CORPUS ACCOUNTING AND COMPOSITION

  • The curated corpus contains 993,920 entries and 1,464.93 audio hours, with a held-out validation set of 6,106 entries; its pools include stereo video-audio, mono-dominant video-audio, and audio-only data.Source: §C.1 · pp. 19–20, Table 12 · p. 19

C.2 SIGNAL-LEVEL STEREO SCREENING AND AMBISONIC DECODING

  • Stereo screening removes silent, phase-corrupted, narrow-stereo, and dual-mono samples, while ambisonic sources are rotated toward the primary horizontal source and decoded at ±45°.Source: §C.2 · p. 20

C.3 MULTIMODAL VERIFICATION AND AUDIO-CENTRIC CAPTIONING

  • Qwen3-Omni scores audiovisual consistency and flags non-diegetic music; captions describe sound attributes while omitting visual appearance, with video-audio captions using both modalities and audio-only captions using audio alone.Source: §C.3 · p. 20

C.4 EVALUATION SET ISOLATION AND LEAKAGE AUDIT

The evaluation sets are checked for separation from training data, and the section specifies objective metrics and a physical-onset protocol. The metrics include stereo-aware measures, while synchronization offsets are better when lower.

C.4 EVALUATION SET ISOLATION AND LEAKAGE AUDIT

  • Interactive evaluation clips are source-group isolated from training and rollout inventories; VGGSound uses its official test split.The 5-s and 10-s Interactive sets contain 4,096 clips from 3,111 unique source groups, with at most two non-overlapping windows per source; the 30-s set has 1,024 distinct held-out sources.Source: §C.4 · p. 20

D.1 OBJECTIVE METRICS

  • FDO uses the mid-channel signal, S-FDO the side-channel signal, and lower DeSync values indicate better synchronization.DeSync is the mean absolute audiovisual synchronization offset in seconds.Source: §D.1 · p. 20

D.2 PHYSICAL ONSET EVALUATION PROTOCOL

  • Greatest Hits onset evaluation uses 244 physical-collision clips with 50 ms pre/post-peak context and a 300 ms matching tolerance.The protocol uses 8-s clips at 22.05 kHz and threshold δ = 0.3.Source: §D.2 · p. 21

D.3 HARDWARE BENCHMARK AND STREAMING LATENCY PROTOCOL

The section reports streaming latency and describes the baseline access modes and stereo-balance metric limitations. It also details the BiasSkill and paired prompt-intervention protocols, including prompt-stratified gains.

D.3 HARDWARE BENCHMARK AND STREAMING LATENCY PROTOCOL

  • 41.21 ms p50 latency per 100 ms chunk yields RTF 0.41; continuous 10 s streaming has RTF 0.466, and prompt updates slightly increase chunk latency.On one NVIDIA H100 GPU at batch size 1 and 15 Euler steps, prompt-update latency is 45.28 ms p50 and 47.40 ms p95.Source: §D.3 · p. 21

D.4 BASELINE REPRODUCTION AND ACCESS MODE ANALYSIS

  • AudioX, ThinkSound, and PrismAudio require full-clip visual context, while V-AURA uses localized windows with non-causal components; WorldSonus uses past and current chunk frames.The bidirectional models are described as unsuitable for streaming or live deployment; V-AURA uses 640 ms strides over a 2.56 s visual window.Source: §D.4 · p. 21

D.5 ITD-BASED METRICS AND LIMITATIONS

  • Production-stereo channel differences do not reliably represent physical arrival-time delays, so the section omits ITD-MSE and FSAD from spatial evaluation.Instead, it reports S-FDO and BiasSkill for stereo energy distribution and left/right channel agreement.Source: §D.5 · p. 21

E STEREO BALANCE AGREEMENT (BIASSKILL)

  • BiasSkill normalizes balanced-accuracy gain over a model-specific permutation null; near-mono generations are counted incorrect rather than removed from the denominator.The permutation null uses shuffled whole-clip pairings, and a score of 0% indicates no better than chance under fixed channel bias.Source: E STEREO BALANCE AGREEMENT (BIASSKILL) · p. 22

F.2 PAIRED COUNTERFACTUAL INTERVENTION PROTOCOL

  • The paired intervention compares switching from P1 to P2 at 5 s against holding P1, with matched model, visual inputs, seeds, solver settings, and guidance scales.The reported results are summarized in Table 15; confidence intervals use 20,000 paired bootstrap resamples per benchmark.Source: §F.2 · p. 23, §F.3 · p. 23

F.3 CONFIDENCE INTERVALS AND PROMPT STRATIFICATION

  • Prompt-intervention gain remains positive across all four prompt-similarity quartiles and increases with prompt divergence.Each quartile contains 1,024 clips; Q1 has the most semantically distinct prompt pairs and Q4 the most similar.Source: §F.3 · p. 23, Table 16 · p. 24

F.1 CLAP IMPLEMENTATION AND SEGMENT ALIGNMENT

CLAP evaluation compares mono 5-s audio segments with their corresponding captions. The user study uses randomized A/B comparisons and records judgments across four evaluation dimensions.

F.1 CLAP IMPLEMENTATION AND SEGMENT ALIGNMENT

  • Audio-caption correspondence is measured with LAION-CLAP cosine similarities on non-overlapping 5 s mono segments, macro-averaged within clips and across the test set.Audio is converted to mono at 48 kHz; embeddings are extracted in FP16 and normalized in FP32.Source: §F.1 · p. 22

G SUBJECTIVE USER STUDY PROTOCOL

  • The user study compares five audio sources on 40 Interactive 10 s clips, with 20 evaluators producing 400 pairwise evaluations.The clips comprise 20 gameplay and 20 real-world scenes; V-AURA is excluded because it is monophonic.Source: G SUBJECTIVE USER STUDY PROTOCOL · p. 24
  • Participants choose A, Tie, or B for spatial alignment, temporal synchronization, semantic alignment, and overall preference.Figure 4 shows the paired-sample interface for gameplay and real-world clips.Source: G SUBJECTIVE USER STUDY PROTOCOL · p. 24, Figure 4 · p. 25
Loading 2610.08760v1…