Source-linked AI summary
Video = World + Event Stream
Lianghua Huang, Zhi-Fan Wu, Yupeng Shi, Wei Wang, Mengyang Feng, Cheng Yu, Chen Liang, Junjie He, Chen-Wei Xie, Yu Liu, Jingren Zhou, Ang Wang, Bang Zhang, Baole Ai, Chongyang Zhong, Jinwei Qi, Kai Zhu, Pandeng Li, Peng Zhang, Wenyuan Zhang, Xinhua Cheng, Yitong Huang, Yun Zheng, Yuxiang Bao, Yuzheng Wang, Zhiwei Lin, Zoubin Bi
TL;DR
Real-time video interaction needs a general account of persistent context and changing events beyond a single target application. Wan-Streamer v0.3 introduces world-plus-event-stream pretraining and full-duplex interaction with open-vocabulary behavior, preserving the prior low-latency operating point.
Problem
The prior training objective targets one talking-and-reacting agent, leaving the underlying general-purpose competence for streaming video insufficiently specified.
Method
Wan-Streamer v0.3 decomposes video into persistent world context and time-varying event streams, pretraining real-time world evolution before specializing it for multimodal speech and open-vocabulary behavior.
Results
640×368 video at 25 FPS, a 160 ms streaming unit, approximately 200 ms model-side response latency, and approximately 550 ms total interaction latency are preserved.
Takeaways & Limitations
The framework supports real-time full-duplex agents that speak and perform synchronized, scene-grounded, open-vocabulary behaviors while retaining low-latency streaming.
Takeaways & Limitations
The paper specializes and evaluates the framework on audio-visual interaction, leaving systematic adaptation and evaluation of other downstream tasks to future work.
Abstract
from arXiv · showhide
We present Wan-Streamer v0.3, which reframes our native-streaming interaction model under a single organizing view: a video is a world plus an event stream. The world is the persistent context in which a video unfolds, including the environment, scene, subjects, ambient acoustic conditions, voice characteristics, and other relatively stable conditions. The event stream is everything that changes over time within that world, including scene or environmental changes, subject behavior, speech, and other sounds. This yields a general-purpose pretraining task over large amounts of real video: given a world and incoming input, predict how the world moves, changes, and responds in real time. The resulting competence can be specialized to a broad family of real-time downstream tasks. We instantiate it on real-time full-duplex audio-visual interaction, where the event stream is the agent's speech together with free-form behavior. Functionally, the model's multimodal understanding process is vision-language-action-like: it maps multimodal user input to language-form speech and behavior actions. Wan-Streamer v0.3 preserves the v0.2 operating point: 640x368 video at 25 FPS, a 160 ms streaming unit, approximately 200 ms model-side response latency, and approximately 550 ms total interaction latency under a 350 ms bidirectional network budget.
1 Introduction
Wan-Streamer v0.3 reframes video as a persistent world plus a time-varying event stream, defining broad pretraining for predicting how worlds evolve in real time. It applies this competence to full-duplex audio-visual interaction with speech and open-vocabulary behavior while preserving the v0.2 operating point.
- World + Event Stream: Video is decomposed into a persistent world context and an event stream of time-varying changes.The world includes scene, subjects, ambient acoustics, and voice characteristics; events include behavior, speech, sounds, and environmental changes.
- General-Purpose Pretraining: Large-scale ordinary-video pretraining learns to predict how a given world moves, changes, and responds unit by unit in real time.The objective produces world knowledge about how scenes plausibly evolve and is intended to transfer across real-time tasks.
- Operating Point: 640×368 video at 25 FPS, a 160 ms streaming unit, approximately 200 ms model-side response latency, and approximately 550 ms total interaction latency are preserved.The total interaction latency uses a 350 ms bidirectional network budget, and the unchanged topology introduces no new latency-critical path.
- Interaction Instantiation: The interaction model maps streaming multimodal user input to speech and open-vocabulary behavior actions in a VLA-like process.The world configures the scene, character, ambient sound, and voice, while the event stream interleaves spoken words with behavior directives that condition synchronized audio-video generation.
2 World-Event Decomposition
The paper decomposes video into a relatively persistent world and a time-varying event stream, representing each as structured context and localized changes. This supports causal next-event prediction as general-purpose pretraining, specialized here for real-time full-duplex audio-visual interaction.
- World and event stream: A video consists of a persistent world containing visual, environmental, subject, acoustic, and voice context plus an event stream of behavior, motion, speech, and sounds.The world remains relatively stable over a clip or interaction, while events unfold within that context.
- World representation: The world uses a task-dependent structured schema with visual, acoustic, and character fields that may be absent or extended.Character records can encode identity, appearance, persona, and visibility.
- Event representation: Events are time-localized records with active intervals, free-form descriptions, and character associations, allowing overlapping changes through multiple records.Descriptions may cover behavior, camera motion, environmental change, speech, or physical sound, including character-independent events.
- Causal prediction: The model predicts each next event and its audio-visual realization from causal history, combining next-token prediction for discrete language and behavior with conditional flow matching for continuous latents.The formulation treats the earlier autoregressive model as the special case where the world is folded into history.
- Pretraining and specialization: Equation (3) defines general-purpose pretraining over diverse video by inferring each clip’s world and aligned events, while this paper specializes the resulting competence to real-time full-duplex audio-visual interaction.Systematic adaptation and evaluation of other downstream interfaces are left for future work.
3 Real-time Interaction
For real-time full-duplex audio-visual interaction, Wan-Streamer represents the persistent world through scene, character, ambient-sound, and voice settings, then maps streaming user inputs to synchronized speech and behavior. Version 0.3 expands behavior from visible listening actions to open-vocabulary, language-described actions grounded in the configured world.
- World context: The persistent world comprises the agent’s scene, character, ambient-sound, and voice-timbre settings.These settings define the environment, identity and persona, background acoustic field, and speaking voice.
- Event stream: Streaming user text, audio, and video drive an event stream whose responses interleave speech with free-form behavior records.Causal decoders render both response types into synchronized audio and video.
- Behavior channel: v0.3 widens the behavior channel from visible listening behaviors to open-vocabulary actions described in natural language.The role-play chat format interleaves spoken words with parenthesized behavior directives.
- Behavior channel: Language-described directives ground actions in the configured world rather than restricting the agent to a fixed action set.Examples include reaching for nearby objects, turning toward sounds, changing posture, and reacting expressively; directives and speech share one token stream.
- Multimodal mapping: Functionally, the model is vision-language-action-like, mapping multimodal user input to speech and behavior tokens that condition joint audio-video generation.Behavior directives use short language tokens and are realized visually through actions such as posture and gesture.
4 Version Comparison
Wan-Streamer v0.3 preserves v0.2’s end-to-end streaming setup, video operating point, latency budget, and serving topology while changing its conceptual framing, pretraining objective, and expressive range. The new framing treats video as a world plus an event stream and trains prediction of how the world streams forward, supporting speech and free-form behavior.
- Changed in v0.3: v0.3 changes the conceptual framing from native streaming to a video-as-world-plus-event-stream view.The world is persistent context, while the event stream contains changes over time, including scene changes, behavior, speech, and other sounds.
- Changed in v0.3: v0.3 changes pretraining to predict how a world streams forward and expands the agent’s expressive range to speech plus free-form behavior.This supports real-time full-duplex audio-visual interaction in which multimodal input maps to language-form speech and behavior actions.
5 Experiments
Wan-Streamer v0.3 maintains the v0.2 streaming operating point while enabling real-time speech with synchronized, open-vocabulary behavior in generated conversations. Its behavior is grounded in the configured scene and consistent with the character’s identity and appearance.
- Latency and runtime protocol: 200 ms model-side latency, 640×368 video at 25 FPS, and approximately 550 ms total interaction latency are maintained under a 350 ms bidirectional network budget.The protocol uses 160 ms user streaming units and the same response boundary as v0.1/v0.2.
- Qualitative behavior observations: Wan-Streamer v0.3 speaks while performing open-vocabulary behavior in real time in generated 640×368 conversations.The observed behaviors include facial expressions, posture changes, sound responses, and interactions with nearby objects.
- Qualitative behavior observations: Behavior actions remain synchronized with speech, grounded in the configured scene, and consistent with the character’s identity and appearance.The passage reports these properties across the observed generated conversations.
6 Conclusion
Wan-Streamer v0.3 reframes native-streaming generation as a decomposition into persistent world context and everything that changes over time. The framework supports general-purpose video pretraining and downstream post-training for real-time full-duplex audio-visual interaction.
- 6 Conclusion: Wan-Streamer v0.3 decomposes native-streaming generation into a persistent world context and an event stream containing everything that changes over time.The world includes persistent context, while the event stream captures temporal changes.
- 6 Conclusion: The decomposition supports general-purpose video pretraining and transfer to roaming, audio-visual interaction, and embodied manipulation.These are identified as downstream transfer targets of the framework.
- 6 Conclusion: The paper instantiates the framework through downstream post-training for real-time full-duplex audio-visual interaction.In this instantiation, VLA-like multimodal understanding maps streaming user input to language-form output and behavior.
Appendix · A Contributions and Acknowledgements · A.1 Core Contributors
The core contributors are Lianghua Huang, Zhi-Fan Wu, Yupeng Shi, Wei Wang, Mengyang Feng, Cheng Yu, Chen Liang, Junjie He, Chen-Wei Xie, Yu Liu, and Jingren Zhou.
- A.1 Core Contributors: Lianghua Huang and Zhi-Fan Wu are listed as core contributors.
- A.1 Core Contributors: Yupeng Shi and Wei Wang are listed as core contributors.
- A.1 Core Contributors: Mengyang Feng and Cheng Yu are listed as core contributors.
- A.1 Core Contributors: Chen Liang and Junjie He are listed as core contributors.
- A.1 Core Contributors: Chen-Wei Xie and Yu Liu are listed as core contributors.
- A.1 Core Contributors: Jingren Zhou is listed among the core contributors.
A.2 Contributors
The paper lists 16 contributors alphabetically by first name, spanning Ang Wang through Zoubin Bi.
- A.2 Contributors: Contributors listed first are Ang Wang, Bang Zhang, Baole Ai, Chongyang Zhong, and Jinwei Qi.The list is ordered alphabetically by first name.
- A.2 Contributors: The middle portion includes Kai Zhu, Pandeng Li, Peng Zhang, Wenyuan Zhang, and Xinhua Cheng.These names continue the alphabetical first-name ordering.
- A.2 Contributors: The remaining contributors are Yitong Huang, Yun Zheng, Yuxiang Bao, Yuzheng Wang, Zhiwei Lin, and Zoubin Bi.These names complete the contributor list.