PaperlayerView exact revision on arXiv

Source-linked AI summary

HelixWorld: A Real-time Interactive Audio-Visual World Model

Lei Ke, Jiahao Pan, Zeyue Tian, Jiaming Wang, Haoyuan Huang, Kam Man Wu, Pengjun Fang, Hongyu Liu, Chenyang Qi, Lin Wang, Ruibin Yuan, Weijia Chen, Fangneng Zhan, Qifeng Chen, Wei Xue, Yike Guo

arXiv:2609.38123v1cs.CV

TL;DR

Interactive world models largely focus on visuals, leaving camera-aligned spatial audio an open challenge. HelixWorld combines curated spatial audio-visual data with a camera- and action-conditioned teacher, then distills it for causal streaming and evaluates spatial-acoustic consistency with HelixBench. It outperforms most open-source silent baselines on visual quality and responsiveness, and surpasses EchoWM and cascaded dubbing pipelines across multimodal measures.

  • Problem

    Interactive world models largely omit audio, and aligning spatial acoustics with dynamic camera motion remains challenging amid data gaps and long-horizon streaming drift.

    Source: §1 · p. 2

  • Method

    HelixWorld trains a bidirectional teacher on curated spatial audio-visual data conditioned on camera trajectories and user actions, then distills it into a causal student using online trajectory supervision.

    Source: §1 · pp. 2–3

  • Results

    HelixWorld outperforms most open-source silent baselines on visual quality and control responsiveness, and surpasses EchoWM and cascaded dubbing pipelines across multimodal measures.

    Source: §5.2 · pp. 7–8

  • Takeaways & Limitations

    HelixBench formalizes spatial-acoustic consistency as a criterion for interactive audio-visual world models and evaluates whether sound fields track camera motion.

    Source: §1 · p. 2, §7 · p. 10

Abstract

from arXiv · show

World simulation is inherently multisensory, demanding synchronized visual and acoustic dynamics in real time. Yet prevailing interactive world models remain strictly silent, focusing exclusively on visual rendering and control while overlooking the acoustic dimension. We present HelixWorld, a real-time interactive audio-visual world model where visual scenes and camera-grounded spatial stereo sound co-evolve natively under user interaction. We curate a high-fidelity spatial audio-visual dataset with true stereo acoustics and metric camera poses, upon which we pre-train a bidirectional teacher conditioned on 6-DoF camera trajectories and user actions. To enable low-latency causal interaction, we distill the teacher into a few-step streaming student via an online trajectory distillation loss, sustaining drift-free joint audio-visual rollouts at 24 FPS on a single GPU. Furthermore, we formalize spatial-acoustic consistency and introduce HelixBench to evaluate whether synthesized sound fields faithfully track dynamic viewpoint motion. Extensive experiments demonstrate that HelixWorld matches state-of-the-art silent world models in visual fidelity and responsiveness, while significantly surpassing existing baselines in camera-aligned spatial-acoustic immersion.

1 Introduction

Interactive world models largely omit sound, and existing approaches struggle to align spatial audio with camera motion under real-time user control. HelixWorld combines camera-conditioned audio-visual generation, causal distillation, curated spatial data, and a benchmark for evaluating that alignment.

1 Introduction

  • Prevailing interactive world models focus almost exclusively on visuals, while spatial-audio alignment under dynamic camera movement remains challenging.The paper identifies missing camera-trajectory data, long-horizon error accumulation, and unstable causal training as obstacles.Source: §1 · p. 2
  • HelixWorld conditions a bidirectional teacher on 6-DoF camera trajectories and user actions, then distills it into a causal student for synchronized generation at 24 FPS.Cross-modal attention binds visual geometry to acoustic features, while online trajectory distillation supervises student velocities from intermediate rollout states.Source: §1 · p. 2
  • The data pipeline removes pseudo-stereo and non-diegetic audio and pairs metric camera poses with decoupled captions for visual scenes, foreground events, and background ambience.The resulting corpus supports bidirectional pre-training and causal streaming distillation.Source: §1 · pp. 2–3
  • HelixBench evaluates 1,015 human-verified interactive cases across five dimensions, including spatial-acoustic consistency between emitter positions and stereo panning.Its spatial criterion measures whether the sound field rotates with camera ego-motion.Source: §1 · pp. 2–3

2 Dataset Construction

The dataset construction pipeline combines real-world, game, and curated open-source footage, filtering multimodal artifacts and annotating camera motion and captions. Its retained corpus supplies synchronized clips with authentic stereo sound and per-frame action annotations.

2 Dataset Construction

  • The construction pipeline addresses datasets' lack of authentic spatial audio by filtering artifacts and annotating metric camera poses, discrete actions, and decoupled captions.These annotations support bidirectional pre-training and causal streaming distillation.Source: §2 · p. 3

2.1 Data Sources

  • Three complementary sources contribute real-world first-person exploration, game footage with camera trajectories and control telemetry, and audio-equipped open-source video subsets.The open-source subsets undergo the same stereo and semantic curation.Source: §2.1 · p. 3

2.2 Curation Pipeline

  • A four-stage, cost-ordered pipeline filters raw footage, standardizes clips, removes semantic artifacts, and then runs camera recovery and captioning in parallel.Camera recovery estimates metric poses for web and real-world footage or uses game-engine ground truth; a clip is retained only when both annotation branches succeed.Source: §2.2 · p. 3, Figure 2 · p. 4
  • 3.0k hours from 4.1k raw hours survive as 2.1M synchronized clips, an overall yield of 73.1%.The resulting corpus provides authentic stereo sound fields and per-frame action annotations for bidirectional pre-training and causal streaming distillation.Source: §2.2 · p. 4

3 Method

HelixWorld progressively trains a bidirectional teacher with camera and action controls, then distills it into a few-step causal student for real-time generation.

3.1 Bidirectional Teacher Training

  • The three-stage teacher training progresses from control-module optimization with the backbone frozen, to video-branch adaptation, then joint audio-visual training.Camera trajectories and user actions enter the video branch and propagate to audio through cross-modal attention.Source: §3.1 · pp. 4–5, Figure 3 · p. 5

3.2 Causal Distillation

  • Causal initialization synchronizes video latents and co-temporal audio tokens into bidirectional temporal blocks, while cross-block attention remains causal.The student is initialized by regressing single-step predictions to the causal teacher’s Probability Flow ODE endpoints.Source: §3.2 · p. 5
  • Online trajectory distillation complements distribution matching to preserve diversity and stabilize autoregressive rollouts.Training stochastically selects between the two losses, with gradients evaluated at the current step and past caches and rollout steps detached.Source: §3.2 · pp. 5–6
  • Long-horizon tuning extends student rollouts segment by segment with a sliding KV cache, retaining sink frames and detaching history.Gradients are computed only on the active segment, bounding memory.Source: §3.2 · p. 6, Figure 4 · p. 6

4 HelixBench: A Benchmark for Audio-Visual World Models

HelixBench evaluates interactive audio-visual rollouts across audio quality, synchronization, semantic alignment, and viewpoint-dependent stereo panning. Its reported comparison favors joint generation over re-dubbing for spatial acoustic alignment.

Benchmark motivation

  • HelixBench addresses the lack of interactive audio-visual evaluation by testing whether sound is realistic, synchronized, prompt-aligned, and responsive to camera motion.Existing world-model benchmarks cover silent video, while audio-visual benchmarks focus on static, non-interactive clips.Source: §4 · p. 6

Benchmark composition and metrics

  • HelixBench contains 1,015 human-verified clips across six scene domains and four subsets, with audio-quality, alignment, synchronization, and stereo-panning measures.The subsets are Navigation, Perspective, Event, and Subject; measures include KL divergence, FAD, ImageBind, CLAP, DeSync, and Spatial.Source: §4 · pp. 6–7

HelixBench results

  • Joint generation achieves superior spatial acoustic alignment to HelixWorld+X re-dubs on HelixBench.HelixWorld+X denotes the paper’s videos re-dubbed using V2A model X.Source: Table 4 · p. 7

5 Experiments

HelixWorld is evaluated against visual, audio, and audio-visual baselines, with ablations testing its conditioning and streaming-training choices. The results cover benchmark performance, extended rollouts, qualitative control, and user preferences.

5.2 Main Results

  • HelixWorld outperforms most open-source silent baselines on visual quality and control responsiveness, and surpasses EchoWM and cascaded dubbing on audio-visual measures.EchoWM retains coarse spatial cues but trails in acoustic quality and semantic binding; video-to-audio models fail to capture emitter locations.Source: §5.2 · pp. 7–8
  • At 768×512 and 24 fps, HelixWorld achieves a steady-state RTF of 0.77 on one NVIDIA H800 GPU, including video and audio decoding.The paper reports that this enables real-time responses to user control.Source: §5.2 · p. 8

5.3 Ablation Studies

  • Progressive fine-tuning reduces both rotation and translation errors in ViPE-recovered trajectories compared with full fine-tuning.The authors conclude that gradually unfreezing branches is essential for learning geometric control while preserving pretrained visual representations.Source: §5.3 · p. 8, Table 5 · p. 8
  • Three-track video, audio, and joint captions consistently yield the highest ImageBind similarity across the tested prompt formats.The three formats are video-only, video-plus-audio, and video-plus-audio-plus-joint captions.Source: §5.3 · p. 8, Table 6 · p. 8

5.4 Qualitative Analysis and User Study

  • Under shared first-frame and control inputs, HelixWorld preserves recognizable scene structure during rising and returning views and a garage motion reversal.The reported examples are a bamboo forest corridor and a red car with nearby pillars.Source: §5.4 · p. 8
  • In blind pairwise tests with audio, HelixWorld wins a majority of votes in every baseline matchup; when muted, it beats three models, ties LingBot v2, and trails EchoWM.The muted comparisons are against Alaya, HY-WorldPlay, SANA, LingBot v2, and EchoWM.Source: §5.4 · p. 8

5.3 Ablation Studies

  • Long-horizon tuning adds consistent, orthogonal gains to both self-forcing variants, and its combination with trajectory distillation reaches the best overall WBench score.The authors describe this tuning as complementary to block-level trajectory guidance for stable interactive rollouts over extended horizons.Source: §5.3 · p. 8, Table 7 · p. 9
  • Online trajectory distillation improves overall WBench performance and mitigates visual drift in 30-second rollouts.Figure 5 shows better preservation of cloud structures and scene landmarks, while Table 8 reports improved DINOv3 and CLIP feature diversity over DMD alone.Source: §5.3 · p. 8, Figure 5 · p. 9

6 Related Work

Prior interactive world models primarily simulate visual dynamics, while audio-visual systems either lack camera-aware control or operate on fixed clips. HelixWorld’s related work positions its contributions around interactive joint generation, causal streaming, and evaluation of spatial sound under viewpoint motion.

Interactive world models

  • Interactive world models accept latent, keyboard, camera-trajectory, or hybrid controls, but prevailing models are silent; EchoWM adds audio without camera-conditioned spatial acoustics.HelixWorld instead conditions a joint audio-visual backbone on continuous camera trajectories and discrete actions.Source: §6 · p. 9

Audio-visual generation

  • Cascaded dubbing lacks camera extrinsics and control intent, while joint audio-visual models co-denoise modalities but remain passive, bidirectional, and fixed-length.HelixWorld brings interactive control and causal streaming to joint models.Source: §6 · p. 9

Causal distillation

  • Video distillation methods address causal streaming, but extending distribution matching to joint audio-visual diffusion can cause mode collapse, color degradation, and acoustic dropouts.Online trajectory distillation anchors the student’s joint velocity field to teacher probability flows; it also improves DINOv3 and CLIP feature diversity over DMD alone.Source: §6 · p. 9, Table 8 · p. 9

Audio-visual evaluation

  • Existing benchmarks either omit audio from action-conditioned video evaluation or test audio-visual correspondence on short fixed-view clips, leaving sound location under listener motion untested.HelixBench formalizes spatial-acoustic consistency alongside temporal, semantic, and dynamic metrics.Source: §6 · p. 9

7 Conclusion

HelixWorld couples interactive visual scenes with spatial stereo sound under dynamic control, using data curation and causal distillation to support synchronized 24 FPS streaming on one GPU. The conclusion frames spatial-acoustic consistency as a criterion for interactive world models and introduces HelixBench to evaluate it.

7 Conclusion

  • HelixWorld’s curation and causal-distillation framework supports synchronized 24 FPS streaming on a single GPU, while HelixBench evaluates spatial-acoustic consistency.The conclusion describes visual scenes and spatial stereo soundscapes as co-evolving under dynamic user control.Source: §7 · p. 10

A Architecture and Conditioning Details

  • Camera conditioning supplements video self-attention, expressing extrinsics relative to the first frame and normalizing intrinsics to the latent grid.Missing or invalid camera tokens and audio tokens receive identity transforms; the camera branch shares Q/K/V projections with base attention.Source: A Architecture and Conditioning Details · p. 16
  • The 81 action classes encode translation and view rotation, with each frame’s action embedding added to the noise-level embedding.Camera and action controls enter the video branch and reach audio through cross-modal attention.Source: A Architecture and Conditioning Details · p. 16

B Training and Streaming Inference Details

The training and inference details specify blockwise audio-visual generation, causal initialization, self-forcing optimization, and bounded streaming history. The appendix also summarizes the scope of the dataset-construction details.

B.1 Temporal Blocks and First-Frame Conditioning

  • A short self-forcing window uses 16 video latents and 127 audio tokens across four blocks, with the conditioning image fixed in the first block.Attention is bidirectional within each block and causal across blocks; historical latents and committed KV tensors are detached while gradients remain in the current block.Source: §B.1 · p. 16

B.2 Causal Initialization

  • Causal initialization uses clean ground-truth history for teacher forcing, then regresses the student onto teacher-forced causal-model endpoints.The ODE teacher uses 50 solver steps; the student input states come from four nonzero knots, and initialization optimizes joint video/audio endpoint MSE.Source: §B.2 · p. 16

B.3 Self-Forcing Optimization

  • Self-forcing starts from the ODE-initialized causal model and trains across a four-block rollout with gradients at one selected denoising step per block.Generator-score and fake-score queries use different sampled noise ranges; with probability 0.1, a generator event uses trajectory distillation instead of a DMD update, anchoring student predictions to a dense teacher integration.Source: §B.3 · pp. 16–17

B.4 Long-Horizon Tuning and Streaming Inference

  • Long-horizon tuning rolls out about one minute of synchronized video and audio, supervising the newest window while detaching earlier latents and KV states.Inference retains the fixed first block, two most recent completed blocks, and current target block, and jointly generates both modalities with four stochastic denoising steps.Source: §B.4 · p. 17

C Dataset Construction and Curation Details

  • The dataset-construction appendix covers domain distribution, privacy, filtering thresholds, perception schema, camera-pose and metric-scale recovery, and caption contracts.These are presented as further details of the curation and annotation pipeline.Source: C Dataset Construction and Curation… · p. 18

C.1 Data Sources, Domain Distribution, and Ethical Considerations

The curation pipeline combines footage from real-world, game, and open-source sources, then filters for authentic stereo, continuous physical scenes, and usable camera motion. It recovers metric camera poses and uses separate visual, audio, and joint captions for supervision.

C.1 Data Sources, Domain Distribution, and Ethical Considerations

  • 1.8 k hours of real-world video, 1.4 k hours of game recordings, and 0.9 k hours of open-source footage form the three-source corpus.Game footage includes native engine audio and ground-truth camera extrinsics and control inputs; identifiable people and private spaces are purged from real-world footage.Source: §C.1 · p. 18

C.2 Signal-Level Filtering and Clip Standardization (Stages 1 & 2)

  • Clips are screened for stereo authenticity and temporal continuity, then standardized to synchronized 121-frame video and stereo audio.Uploads with inter-channel difference δ < 0.01 are discarded; short or rapidly cut shots are also rejected.Source: §C.2 · p. 19

C.3 Semantic Filtering with Audio-Visual MLLM (Stage 3)

  • Qwen3-Omni-30B retains clips only when they show a dynamic 3D physical environment with natural diegetic sound.Off-screen environmental sounds are retained, while background music, voiceover, and artificial sound effects are pruned.Source: §C.3 · p. 19

C.4 Camera Pose Estimation and Metric Scale Recovery (Stage 4)

  • VGGT-Omega has the lowest rotation error, 0.234°, among the evaluated camera estimators, and metric scale recovery grounds camera translations in physical units.Camera motions are gated against kinematic bounds before being quantized into the 81-class action vocabulary.Source: §C.4 · pp. 20–21

C.5 Decoupled Multimodal Captioning (Stage 4)

  • Three independent caption streams describe visuals, audio, and joint audio-visual events, eliminating textual cross-talk while preserving cross-modal synchronization supervision.The audio caption avoids directional terms so stereo panning remains governed by geometric control.Source: §C.5 · p. 21

D HelixBench and Efficiency Evaluation

HelixBench evaluates audio quality, cross-modal alignment, and spatial audio–visual agreement across human-verified cases. In spatial-threshold sweeps, HelixWorld ranks first among the world models, and its native audio outperforms the post-hoc replacements across all tested settings.

D.1 Case Selection

  • The 1,015 benchmark cases are organized by scene and interaction labels, while metric eligibility is determined separately for each applicable subset.The spatial subset retains cases with a single visible, localizable dominant source; sources are isolated from training at the whole-video level.Source: §D.1 · pp. 21–22

D.2 Audio Quality

  • Audio quality is measured by reference-to-generated event-distribution KL divergence and pooled-embedding Fréchet Audio Distance, with lower values indicating closer agreement.KL uses the first five seconds of each case; FAD compares pooled VGGish feature distributions.Source: §D.2 · p. 22

D.3 Cross-Modal Alignment

  • Cross-modal alignment uses DeSync for estimated temporal misalignment and ImageBind and CLAP for semantic agreement, with higher similarity scores indicating stronger agreement.CLAP compares generated audio with the input audio-visual caption.Source: §D.3 · p. 22

D.4 Spatial Audio–Visual Agreement

  • The spatial score compares detected source position in generated video with stereo pan, rewarding same-side audio and penalizing opposite-side audio for eligible lateral samples.Visual-center samples are excluded, and always-centered audio scores zero.Source: §D.4 · pp. 22–23

D.5 Post-hoc Video-to-Audio Comparison

  • The post-hoc dubbing comparison replaces the audio on the same HelixWorld videos with PrismAudio, ThinkSound, or AudioX, using native audio as the matched reference.The protocol covers all 1,015 test cases with shared videos and captions across conditions.Source: §D.5 · p. 23

D.6 Spatial Threshold Sensitivity

  • HelixWorld ranks first among four world models across 20 spatial-threshold settings, and its native audio outperforms all three video-to-audio replacements throughout the grid.The world-model sweep uses Perspective candidates; the fixed-video comparison shares detections and eligibility across audio conditions.Source: §D.6 · p. 23, Table 12 · p. 24, Table 13 · p. 24

D.7 Inference Efficiency

  • Inference timing uses a single NVIDIA H800 and includes in-loop geometry processing, CPU offload, decoding, and frame transfer, while excluding loading and encoding.Streaming real-time factor measures steady-state generation after delivery of the first block; values below one indicate faster-than-real-time throughput.Source: §D.7 · pp. 23, 25

E Additional Trajectory Distillation Results

Trajectory distillation increases conditional visual diversity over DMD-only self-forcing, and additional examples compare visual drift with and without distillation.

E.1 Conditional Visual Diversity

  • Conditional diversity averages cosine distances between generations at matched times, excluding the fixed first frame and giving each source equal weight.The calculation uses DINOv3 and CLIP features from matched frames across generations of the same input.Source: §E.1 · p. 25
  • 21.7% higher DINOv3 diversity and 13.5% higher CLIP diversity are reported for self-forcing with online trajectory distillation versus DMD-only self-forcing.The comparisons are reported in Table 8.Source: §E.1 · p. 25

E.2 Additional Qualitative Comparisons

  • Figure 9 provides additional visual comparisons of DMD-only self-forcing against self-forcing with trajectory distillation, with reduced visual drift noted for the latter.The figure pairs DMD-only examples above trajectory-distillation examples.Source: §E.2 · p. 25

F User Study Protocol

The user study compares audio-enabled and muted-video outputs across model pairs using blinded, randomized A/B presentations. Additional comparisons illustrate audio–visual timing and stereo correspondence, including cases where AudioX produces a transient during a visible pause.

F User Study Protocol

  • The blinded, randomized study gathers 212 audio-enabled and 190 muted-video responses across 20 scenes in each setting.Audio-enabled comparisons include EchoWM and three post-hoc dubbing baselines; muted-video comparisons include five world models.Source: F User Study Protocol · p. 25

G Additional Audio-Visual Comparisons

  • Golf-swing examples show HelixWorld’s acoustic transients coinciding with visible strikes, while the drumming comparison shows AudioX producing a transient during a visible pause.HelixWorld’s drumming transients subside during the pause and resume with the motion.Source: Figure 10 · p. 27, Figure 11 · p. 28
  • Stereo examples show HelixWorld maintaining stronger right-channel energy for a rightward-moving fountain and shifting channel balance from left to right as a vehicle passes.The comparisons use left and right audio channels paired with the corresponding video motion.Source: Figure 12 · p. 29, Figure 13 · p. 30

Related papers

Loading 2609.38123v1…