PaperlayerView exact revision on arXiv

Source-linked AI summary

EchoWM: Open and Enterable Omnimodal World Models

Songchun Zhang, Yaowei Li, Junhao Zhuang, Weiyang Jin, Haoyu Wang, Xin Lu, Yilang Sun, Shiyi Zhang, Haoran Li, Xiaoxiao Ma, Yuming Li, Yijun Liu, Yaofeng Su, Yanwen Ma, Haoyu Wu, Zihan Su, Yue Ma, Lvmin Zhang, Haoyang Huang, Zeyue Xue, Anyi Rao, Nan Duan

arXiv:2608.23189v1cs.CV

TL;DR

Existing world models largely lack continuous, multimodal interaction that remains synchronized over long horizons. EchoWM addresses this with camera-intent conditioning, a calibrated relative 6-DoF trajectory, complementary data, and progressive autoregressive training, achieving strong benchmark performance while supporting varied first- and third-person scenes. Its scope remains limited to navigation and viewpoint-related actions, without explicit arbitrary actor intent or persistent 3D memory.

  • Problem

    Existing systems are often silent or restricted to rigid, subject-specific action spaces, leaving continuous multimodal participation across general and game scenes open.

  • Method

    EchoWM uses camera intent, a shared metric-scale relative 6-DoF trajectory, complementary world data, progressive training, and autoregressive post-training for enterable audio-visual generation.

  • Results

    EchoWM ranks first on WBench Navigation and achieves top-tier visual quality on SANA-WM-Bench while maintaining visual and control consistency over long-horizon rollouts.

  • Takeaways & Limitations

    The framework supports synchronized video, environmental sound, and speech during continuous first- and third-person navigation across diverse scenes and subjects.

  • Takeaways & Limitations

    EchoWM focuses on relative 6-DoF navigation and does not explicitly represent arbitrary actor intent or maintain persistent 3D memory.

Abstract

from arXiv · show

We present EchoWM, an omnimodal world model for enterable generative media that responds to continuous navigation while jointly generating 720p video, environmental sound, music and speech. We organize interaction around camera intent: in first-person scenes, it specifies observer motion, while in third-person scenes, camera--character dynamics are learned from data without view-specific controllers. Discrete commands and continuous poses are mapped to a shared metric-scale relative 6-DoF trajectory, with dataset-level calibration preserving motion magnitude across heterogeneous data. To jointly learn audio-visual generation and trajectory control, we construct a complementary data engine and adopt progressive training followed by autoregressive post-training for long-horizon generation. Extensive evaluations show that \model achieves strong trajectory following and high visual quality on public world-model benchmarks, supporting both first- and third-person interaction across varied subjects, and maintaining synchronized environmental sound and speech over long-horizon generation.

1 Introduction

EchoWM defines an enterable omnimodal world model for continuous navigation with synchronized video, environmental sound, and speech across first- and third-person scenes. It unifies camera intent, complementary data, and progressive training to support controllable, long-horizon generation.

  • EchoWM generates 720p video, environmental audio, and speech from navigation inputs, covering diverse scenes and subjects while achieving strong public-benchmark performance.It ranks first on WBench Navigation and reaches top-tier visual quality on SANA-WM-Bench across simple and challenging trajectory splits.
  • EchoWM combines continuous user navigation with native joint generation of video, environmental sound, and speech in an enterable generative-media setting.
  • The model supports first- and third-person interaction and synchronized multi-turn continuation beyond a single rollout.
  • The interface maps discrete commands and continuous camera poses to a shared relative 6-DoF trajectory with dataset-level metric calibration.The same camera-intent interface supports first-person observer motion and third-person camera–character evolution without view-specific controllers.
  • A capability-aligned data engine combines gameplay, simulation, and Internet video into AV-rich, control-clean, and balanced mixtures.These mixtures assign audio-visual generation and trajectory-following supervision to the subsets where each signal is most reliable.
  • Progressive training first adapts joint audio-visual generation, then learns trajectory conditioning, jointly fine-tunes both components, and uses autoregressive post-training for long-horizon generation.

2 Related Work

Related work spans action-conditioned world models, camera-trajectory-controlled video generation, joint audio-visual synthesis, and autoregressive methods for causal long-horizon generation.

  • World models have progressed from compact latent dynamics and action-conditioned simulation toward open-ended interactive video generation with broader fidelity, domains, and horizons.
  • Action-conditioned systems support navigation and character control, but their action semantics are typically tied to particular controllers or environments.
  • Joint audio-visual models generate visual and acoustic content together, extending earlier video-to-audio pipelines.
  • Causal and autoregressive approaches use self-generated rollouts, bounded or persistent context, and retrieval-based memory to support longer interactive generation.

3 World Data Engine

EchoWM’s world-data engine combines complementary game, simulation, and Internet sources to supply audio-visual richness, interactive motion, and reliable metric geometry. Separate processing paths and stage-aligned mixtures preserve the signals needed for synchronized generation and trajectory control.

  • Complementary Data Sources: Four complementary sources provide diverse appearance, natural audio, interactive motion, and reliable camera geometry for enterable-world training.The sources are internally collected gameplay, human-played Internet gameplay, Unreal Engine simulation, and general Internet video.
  • Stage-Aligned Mixtures: The processed data form AV-rich, control-clean, and balanced mixtures aligned respectively with AV-CPT, Action-SFT, and Joint-FT.Pose estimation is separated from trajectory quality control, and ViPE combines complementary geometric backends to optimize temporally consistent metric-scale poses and intrinsics.
  • Complementary Data Sources: Internally collected gameplay supplies first- and third-person trajectories, explicit control inputs, and native dialogue, music, ambience, and interaction sounds.Programmatic scripts improve action–observation correspondence, while the native mix retains sounds such as footsteps, vehicles, and collisions.
  • Complementary Data Sources: Human-played Internet gameplay adds natural control timing, scene reactions, in-game dialogue, player commentary, and diverse human speech despite lacking native action logs.Continuous clips with reliable recovered trajectories support control training, while the broader pool strengthens audio-visual and speech modeling.
  • Complementary Data Sources: Unreal Engine simulation contributes ground-truth metric poses, synchronized action logs, and controlled translation and rotation coverage, but lacks natural audio.Engine physics preserves realized motion under collisions and environmental constraints; multiple first- and third-person views are exported with poses and metadata.
  • Complementary Data Sources: General Internet video broadens visual and acoustic diversity with realistic appearance, environmental sound, speech, and camera–subject relationships, primarily strengthening the audio-visual prior.Only a smaller subset with successful metric pose recovery contributes to trajectory-conditioned training.
  • Temporal Processing: Separate audio-visual and geometry paths use short clips for appearance and sound, but preserve approximately one-minute windows for shared metric reconstruction before slicing.Long-window reconstruction gives neighboring clips a common coordinate system and broader temporal support for pose optimization.

4 Method

The method overview organizes EchoWM around shared inputs, trajectory construction, a joint audio-visual backbone, and synchronized outputs. The interface connects user controls to the model’s multimodal generation pipeline.

  • EchoWM’s method is organized into inputs, shared camera-intent interface, trajectory construction, joint audio-visual backbone, and synchronized outputs.

4.1 Overview and Problem Formulation

EchoWM takes media context, structured text, and user controls, converts controls into calibrated relative 6-DoF camera trajectories, and jointly generates synchronized video and audio. Its architecture injects trajectory conditioning into the video backbone and uses progressive plus autoregressive training for continuation.

  • The model jointly generates synchronized video and audio from optional media context, structured text, and a user control sequence while following requested camera motion.Media context may be empty, a reference observation, or a synchronized audio-visual prefix.
  • User controls are converted into a calibrated relative 6-DoF camera trajectory that specifies subsequent camera motion separately from persistent scene descriptions.Static text describes scene, style, subject, and initial viewpoint, while the trajectory explicitly describes camera evolution.
  • EchoWM extends a pretrained joint audio-visual diffusion transformer with metric trajectory conditioning and autoregressive audio-visual generation.Heterogeneous controls and poses share dataset-level translation calibration, and a UCPE-based camera branch is shared across first- and third-person data.
  • Progressive training adapts the audio-visual backbone, learns trajectory conditioning with the backbone frozen, jointly fine-tunes both groups, and then adapts generation to its own history.The trajectory is injected into the video backbone through a UCPE-based camera branch.

4.2 Metric Camera-Intent Conditioning

EchoWM represents navigation as a shared relative 6-DoF camera trajectory, allowing discrete commands and continuous poses to control first- and third-person viewpoint evolution. Dataset-level calibration preserves comparable translation magnitude across heterogeneous sources, while relative camera geometry conditions generation.

  • Unified Camera-Intent Interface: In first-person scenes, the trajectory specifies observer ego-motion; in third-person scenes, learned camera–character coupling determines associated character motion.The model uses no explicit character trajectory, controller state, or camera-rig parameters.
  • Unified Camera-Intent Interface: Camera trajectories broaden training coverage, avoid ambiguous controller-dependent inverse mappings, and preserve continuous translation and rotation beyond discrete keyboard vocabularies.They can be recovered from more videos than frame-aligned action logs and retain vertical motion, roll, magnitude, and translation–rotation coupling.
  • Unified Camera-Intent Interface: A shared relative 6-DoF trajectory converts discrete commands and continuous camera poses into a common geometric conditioning format.This separates user-facing controls from the trajectory representation consumed by the generator.
  • Global Translation Scale Calibration: A single robust dataset-level translation scale aligns physical displacement across heterogeneous sources without per-clip normalization or clipping larger trajectories.The scale is estimated from training trajectories, and retained trajectories have maximum displacement no greater than the global scale.
  • Global Translation Scale Calibration: The same calibrated scale is used during training and inference, while rotations remain unchanged, preserving displacement differences and avoiding continuation-boundary changes in motion speed.A common temporal sampling rate further supports comparable clip-level translation extent.
  • Relative Trajectory Conditioning: UCPE injects pairwise relative ray geometry into attention, making camera relationships invariant to global rigid coordinate changes.Unlike absolute ray representations that depend on the world origin, the camera branch uses relative ray transforms and is shared across first- and third-person data.

4.3 Progressive Audio-Visual Control Training

EchoWM separates audio-visual adaptation from trajectory-control learning through progressive training on complementary data mixtures, then jointly fine-tunes both capabilities. The curriculum uses autoregressive exposure to self-generated histories to support stable long-horizon interactive generation.

  • Data and Objective Separation: Complementary data mixtures address the mismatch between AV-rich examples with narrower motion coverage and control-clean examples with more reliable trajectories.The training stages assign these sources to the capabilities they supervise most directly.
  • Progressive Training: AV-CPT first adapts the complete backbone on AV-rich data to establish visual dynamics and acoustic priors.The stage retains structured annotations including narrative, speech, and sound.
  • Progressive Training: Action-SFT then freezes the audio-visual backbone and trains only the camera-conditioning branch on control-clean trajectories.Static annotations and the trajectory specify camera evolution, while the audio loss is disabled.
  • Progressive Training: The trajectory is a conditioning signal rather than a prediction target, and no explicit action-to-audio objective is introduced during Action-SFT.This keeps trajectory responsiveness separate from audio-visual generation supervision at that stage.
  • Progressive Training: Joint-FT updates the backbone and trajectory branch together on a balanced high-quality subset where audio-visual signals and camera trajectories are reliable.A reduced learning rate limits disruption to capabilities learned in earlier stages.

4.4 Streaming Audio-Visual Post-Training

EchoWM is progressively adapted from a bidirectional audio-visual diffusion model into a causal, chunk-wise autoregressive generator for synchronized streaming. Teacher forcing establishes causal dependencies, while Self-Gradient Forcing exposes the model to self-generated histories and supports few-step generation.

  • Audio-Visual Teacher Forcing: The model is transformed into a causal streaming generator that produces synchronized audio-video chunks with low latency.The transition converts bidirectional temporal attention to causal computation and compresses iterative denoising into a few-step sampler.
  • Audio-Visual Teacher Forcing: Aligned video and audio chunks share a diffusion noise level, placing both modalities at the same diffusion stage.Each modality is divided into temporally aligned macro-chunks, although their token counts may differ.
  • Audio-Visual Teacher Forcing: The causal attention mask lets noisy chunks attend to preceding clean chunks and current noisy tokens, while masking clean targets and future chunks.The same causal constraint is applied to video self-attention, audio self-attention, and both directions of audio-video cross-attention.
  • Audio-Visual Teacher Forcing: Trajectory conditioning is chunk-causal: chunk i uses only the trajectory available through the current chunk.The model jointly predicts audio and video flow velocities in one forward pass while retaining the pretrained flow-matching objective.
  • Self-Gradient Forcing: Self-Gradient Forcing addresses the mismatch between ground-truth teacher-forcing histories and self-generated inference histories while enabling few-step generation.It combines autoregressive self-rollout, differentiable context reconstruction, and distribution matching under self-generated histories.
  • Self-Gradient Forcing: Short-horizon audio-visual SGF exposes the generator to its own histories, restores supervision through reconstructed causal context, and establishes few-step generation.The resulting model initializes long-horizon streaming training.

4.5 Interactive Inference

Interactive inference supports both bidirectional multi-turn continuation and causal streaming. Both modes use trajectory-conditioned synchronized audio-video context, but they differ in how historical context is managed.

  • Bidirectional multi-turn continuation: Bidirectional continuation reuses synchronized video and audio from the preceding segment as clean context for the next turn.Each turn may specify a different navigation trajectory, and the generated segment becomes tail context for the following turn.
  • Inference modes: The two inference modes share the trajectory interface and model parameterization but differ in historical-context management.Bidirectional continuation decodes a tail for re-encoding, whereas causal streaming retains a persistent cache.
  • Causal streaming inference: Causal streaming maintains persistent video, audio, and camera-attention KV caches with sink-plus-local context windows.The sink preserves a long-term prefix while the local window stores recent context; old local tokens are removed and temporal RoPE is rebased.

5 Evaluation

EchoWM is evaluated on complementary benchmarks covering interactive quality, controllability, consistency, and long-horizon trajectory following. It performs strongly across quantitative, qualitative, human, and cross-domain evaluations, while long rollouts reveal accumulated pose drift.

  • Evaluation setup: WBench Navigation evaluates visual quality, setting adherence, interaction, consistency, and physical plausibility across first- and third-person navigation.SANA-WM-Bench focuses on short- and long-horizon camera-trajectory following and visual persistence.
  • WBench Navigation: 81.7 overall Average is achieved by undistilled EchoWM on WBench Navigation, while EchoWM-Flash reaches 81.0 and the highest Interaction score of 87.9.EchoWM also records a Consistency score of 89.8; EchoWM-Flash retains most overall capability after four-step causal distillation.
  • SANA-WM-Bench: EchoWM achieves the highest 241-frame VBench Overall on both Simple and Hard splits, with 83.91 and 83.96, respectively.It has the lowest translation and CMC errors on Simple trajectories, while SANA-WM has lower pose errors on Hard trajectories.
  • Long-horizon evaluation: 12.05° rotation error is reached on the Hard 961-frame split, indicating accumulated pose drift during long-horizon generation.On the Simple 961-frame split, EchoWM achieves VBench Overall 81.36, the lowest translation and CMC errors, and revisit PSNR 15.10 dB.
  • Causal comparison: EchoWM-Flash achieves the highest 961-frame VBench Overall on both Simple and Hard splits, with 80.13 and 81.06, respectively.It also has the strongest trajectory accuracy and revisit consistency among evaluated causal methods, with only moderate degradation relative to the undistilled model.
  • Qualitative evaluation: EchoWM-Flash better preserves viewpoint, scene layout, subject identity, visual style, and geometric structure than competing causal models during long rollouts.Qualitative WBench comparisons report better action adherence and multi-turn consistency.
  • Human evaluation: EchoWM receives 46.50% of overall-preference votes against LingBot-World-v2 and 63.13% against HappyOyster.It leads LingBot-World-v2 in semantic following, spatial-temporal consistency, and visual aesthetics.
  • Generalization and memory: EchoWM maintains coherent viewpoint evolution and major scene or subject structure on unseen 2D/2.5D game and robot domains.Additional qualitative evidence shows visual memory across revisited views and more predictable displacement under fixed global translation scale.

6 Discussion

The discussion frames EchoWM around observation-centered interaction, shared world-level trajectories, explicit scale and viewpoint semantics, joint audio generation, and stage-specific training signals. Together, these design principles connect enterable interaction with evolving synchronized audio-visual generation.

  • Center interaction on observation intent: Camera intent provides one interaction semantics for first-person observer motion and third-person camera-character evolution.The model learns plausible camera-character coupling from data rather than using separate viewpoint-specific control semantics.
  • Represent interaction at the world level: Mapping discrete and continuous inputs to a shared relative trajectory avoids binding navigation to a particular controller or action encoder.This principle applies to navigation and viewpoint-related actions, not arbitrary actor behavior.
  • Make scale and viewpoint semantics explicit: Dataset-level metric scale preserves relative displacement across sources and avoids normalization-induced speed jumps.Viewpoint context is also needed because geometry alone does not specify whether rotation is egocentric, character-following, or orbital.
  • Keep audio inside the generated world: Environmental sound and speech remain part of the native joint audio-visual process while trajectory conditioning enters only the video stream.Joint-FT optimizes this process on trajectory-conditioned examples without an explicit action-to-sound controller.
  • Match each training stage to its cleanest signal: The training curriculum separates audio-visual prior acquisition from navigation acquisition before allowing the two capabilities to co-adapt.AV-CPT uses audio- and speech-rich clips, Action-SFT isolates control on clean motion, and Joint-FT consolidates both on balanced data.
  • Overall design principles: The data engine, shared interaction condition, joint backbone, and continuation process turn individual generations into an evolving audio-visual experience.The discussion presents these components as complementary principles extending beyond the particular backbone.

7 Limitations

EchoWM’s interaction is limited to navigation and viewpoint-related actions represented by relative 6-DoF trajectories, rather than arbitrary actor behavior or game-engine dynamics. Its persistent-world consistency and broader trajectory generalization remain constrained.

  • EchoWM supports navigation and viewpoint-related actions represented by relative 6-DoF trajectories, but not arbitrary behaviors such as jumping, attacking, manipulation, or robot commands.
  • The model does not implement conventional game-engine rules, collision state, or deterministic transitions.
  • Without explicit persistent 3D memory, geometry, subject identity, world state, and audio can drift across repeated continuation turns.
  • Generalization to substantially larger translations, higher speeds, or unusual rotations is not established, and the usable interaction range is restricted by the global trajectory filter.
  • Pose estimates from Internet and gameplay video remain imperfect after filtering and may correlate with scene content or motion blur.

8 Conclusion

EchoWM presents an enterable omnimodal world model that unifies camera-intent navigation with jointly generated audiovisual media across general and game scenes. The resulting system supports first- and third-person interaction, synchronized audio-visual output, and multi-turn continuation.

  • EchoWM bridges generative media and interactive world modeling across general and game scenes using camera intent as a unified interaction perspective.
  • A complementary gameplay–simulation–Internet data engine supplies diverse supervision, while scale-consistent trajectory conditioning aligns navigation across heterogeneous data.
  • A progressive audio-visual-to-action curriculum first learns audiovisual dynamics, then control responsiveness, and finally consolidates both capabilities on jointly high-quality data.
  • The resulting model supports first- and third-person interaction, native joint audio-visual output, and multi-turn continuation in one system.
  • EchoWM suggests a path from generative content that is only watched toward omnimodal worlds that can be continuously entered and influenced.
Loading 2608.23189v1…