Source-linked AI summary
Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models
Lianghua Huang, Zhi-Fan Wu, Wei Wang, Yupeng Shi, Mengyang Feng, Junjie He, Chen-Wei Xie, Yu Liu, Jingren Zhou, Ang Wang, Bang Zhang, Baole Ai, Chen Liang, Cheng Yu, Chongyang Zhong, Jinwei Qi, Kai Zhu, Pandeng Li, Peng Zhang, Wenyuan Zhang, Xinhua Cheng, Yitong Huang, Yun Zheng, Yuzheng Wang, Zoubin Bi
TL;DR
Real-time audio-visual interaction requires causally aligned, full-duplex perception and generation across modalities rather than simple alternation or cascaded processing. Wan-Streamer unifies these capabilities in one native-streaming Transformer, achieving approximately 550 ms total interaction latency with a 350 ms bidirectional network budget.
Problem
Real-time audio-visual interaction requires causally aligned, overlapping perception and generation across modalities in a full-duplex process.
Method
Wan-Streamer models language, audio, and video inputs and outputs as one causal stream in a single Transformer without external modality-specific generation modules.
Results
Approximately 550 ms total interaction latency is achieved with a 350 ms bidirectional network budget, including approximately 200 ms model-side latency.
Takeaways & Limitations
Wan-Streamer demonstrates sub-second, unified full-duplex text, audio, and video interaction with synchronized responses and full-history context.
Abstract
from arXiv · showhide
We present Wan-Streamer, a native-streaming, end-to-end interactive foundation model designed from the ground up for real-time, low-latency, full-duplex audio-visual interaction. Wan-Streamer seamlessly models language, audio, and video as both input and output within a single Transformer, where the sequence is represented as interleaved visual, audio, and text input tokens together with visual, audio, and text output tokens, coordinated by block-causal attention for incremental streaming. Unlike cascaded interactive systems that rely on separate VAD, ASR, language, TTS, audio-driven animation, or video-generation modules, Wan-Streamer does not rely on external language, speech, avatar, or video-generation modules: perception, reasoning, generation, response timing, turn management, and cross-modal synchronization are learned jointly within one unified model, reducing pipeline latency and error accumulation. To support natural audio-visual responsiveness, we redesign the entire stack around streamability, including causal encoders, causal decoders, block-causal attention, and low-latency multimodal token scheduling, enabling streaming units as short as 160 ms at 25 fps. Wan-Streamer achieves approximately 200 ms model-side response latency and approximately 550 ms total interaction latency when combined with 350 ms bidirectional network latency, supporting sub-second duplex audio-visual communication. These results position Wan-Streamer as a unified, end-to-end, multimodal interactive foundation model for low-latency streaming interaction.
1 Introduction
Wan-Streamer is introduced as a native-streaming, end-to-end foundation model for full-duplex audio-visual interaction, unifying language, audio, and video inputs and outputs within one Transformer. Its causal architecture and thinker-performer inference schedule support incremental multimodal generation with approximately 200 ms model-side response latency and approximately 550 ms total interaction latency.
- Motivation: Human interaction is inherently streaming and full-duplex, with perception and expression overlapping as people watch, listen, speak, gesture, react, pause, and interrupt.This motivates systems that can respond continuously across audio-visual timescales rather than alternating between completed perception and response.
- Wan-Streamer: Wan-Streamer models language, audio, and video as both inputs and outputs within a single Transformer without external language, speech, animation, or video-generation modules.The model is designed from the ground up for native streaming and end-to-end real-time full-duplex audio-visual interaction.
- Limitations of existing pipelines: Separately trained multimodal pipelines introduce module-boundary waiting, recognition and synchronization errors, and difficulties learning response timing, turn management, identity preservation, and long-horizon consistency jointly.Text is often used as a hidden intermediate representation between components even when the interface appears multimodal.
- Architecture: Its fully causal multimodal architecture combines causal audio and video VAEs, causal encoders and decoders, block-causal attention, and full-history autoregressive streaming.The streaming contract requires causal operation, immediate use of newly observed units, and emission of generated units back into the interaction history.
- Inference and latency: Approximately 200 ms model-side response latency and approximately 550 ms total interaction latency are achieved by overlapping understanding and generation through KV-cache exchange in a thinker-performer inference system.The thinker processes current observations and updates the language state, while the performer generates the next audio-visual latent unit.
2 Method
Wan-Streamer represents interaction as a continuous causal stream, jointly predicting language tokens and audio-video latents from complete multimodal history. It trains through independent-task pretraining, end-to-end interaction training, and low-latency distillation, then deploys overlapped thinker-performer inference.
- Causal streaming formulation: At each streaming unit, the model predicts the next response from the complete causal history of user observations and prior agent responses.Generated response units are appended with corresponding observations and become context for subsequent units.
- Causal streaming formulation: Language responses use next-token cross-entropy, while audio and video responses use conditional flow matching in continuous latent spaces.The unified diffusion transformer estimates coupled audio-video velocity fields from clean causal context and noise level τ.
- Training: Training proceeds through independent-task pretraining, duplex end-to-end interaction training, and distillation into an efficient low-latency streaming student.Rolling distillation trains the student on its own generated history to reduce long-horizon degradation under realistic rollout conditions.
- Streaming inference: Deployment separates the thinker from the performer while retaining end-to-end training, assigning multimodal encoding, state updates, caching, and decoding to the thinker and latent generation to the performer.The schedule overlaps current observation processing, previous-response decoding, cache and latent transfer, and next-response denoising across streaming units.
- Streaming inference: 160 ms streaming units define the real-time throughput condition: performer wall time plus small KV-cache and latent communication overhead must fit within one unit.Per-frame throughput is determined mainly by performer wall time after overlap.
3 Experiments
Wan-Streamer is evaluated using separate model-side and total interaction latency boundaries, while comparisons distinguish full-loop response from component-level runtime. It also demonstrates unified audio-visual response generation, continuous non-verbal behavior, and learned full-duplex interaction.
- Latency and runtime comparison: Comparisons separate user-perceived signal-to-signal latency from model-internal, first-packet, API, FPS, first-frame, chunk, and audio-to-visual measurements.The paper emphasizes that raw speed alone is insufficient when systems cover different interaction paths.
- Latency and runtime comparison: Wan-Streamer’s 550 ms total latency covers text I/O, user audio-video perception, response timing, speech generation, and 25 FPS visual expression in one end-to-end model.This contrasts with component metrics that may omit upstream dialogue, speech, perception, or synchronized visual output.
- Naturalness: During idle and listening intervals, Wan-Streamer continuously generates identity-preserving behavior including gaze, posture, breathing, facial motion, nods, and micro-expressions.These visible behaviors remain temporally coupled to the streaming history rather than collapsing into a frozen portrait.
- Interruption and proactive speaking: Wan-Streamer learns full-duplex turn management from interleaved interaction data, modeling pauses, overlaps, interruptions, yielding, resumption, and continued user observation during generation.User inputs and agent outputs across text, audio, and video share the same causal timeline.
4 Related Works
Related work has advanced full-duplex speech, streaming audio-driven avatars, and unified audio-visual agents. Wan-Streamer extends the end-to-end direction by jointly modeling language, audio, and video on both sides of interaction within one causal Transformer process.
- Full-duplex spoken dialogue: Full-duplex spoken-dialogue models replace turn-based pipelines with shared streaming timelines for listening and speaking.Moshi uses parallel user and assistant speech streams, while OmniFlatten and SALM-Duplex pursue end-to-end speech-text or codec-based dialogue modeling.
- Interactive digital humans and audio-visual avatars: Audio-driven avatar research has progressed from portrait and talking-head synthesis toward real-time full-body and long-duration character animation.TalkingMachines, StreamAvatar, LiveTalk, Hallo-Live, and OmniForcing target lower latency and improved temporal consistency in audio-visual generation.
- End-to-end audio-visual interaction: A smaller body of work explores unified audio-visual agents, including Body of Her, which integrates audio and visual inputs with speech, behavior, idling, response, and manipulation.FlowAct-R1 and related humanoid video models also indicate a movement toward action-level interactive systems.
- End-to-end audio-visual interaction: Wan-Streamer models language, audio, and video on both input and output sides using a single Transformer with fully causal encoders, decoders, and full-history streaming inference.This unifies perception, reasoning, speaking, visible listening, interruption handling, and synchronized audio-visual generation as one native full-duplex process rather than a cascade.
5 Conclusion
Wan-Streamer is presented as a native-streaming, end-to-end foundation model for real-time full-duplex text, audio, and video interaction. It unifies multimodal inputs and outputs in one causal stream processed by a single Transformer, supported by causal components and block-causal attention.
- Wan-Streamer is a native-streaming, end-to-end foundation model for real-time full-duplex text, audio, and video interaction.
- Unlike cascaded systems, Wan-Streamer represents user inputs and agent outputs across all modalities as one causal stream processed by a single Transformer.
- Fully causal audio and video VAEs, causal encoders and decoders, and a block-causal Transformer support perception and synchronized audio-visual response generation.
Appendix · A Contributions and Acknowledgements · A.1 Core Contributors
The appendix’s core-contributor section lists nine individuals associated with the paper’s contributions and acknowledgements.
- Appendix: Lianghua Huang, Zhi-Fan Wu, and Wei Wang are listed as core contributors.
- A Contributions and Acknowledgements: Yupeng Shi and Mengyang Feng are listed among the core contributors.
- A Contributions and Acknowledgements: Junjie He and Chen-Wei Xie are included in the contributor list.
- A.1 Core Contributors: Yu Liu and Jingren Zhou complete the named core-contributor list.
- A.1 Core Contributors: The listed core contributors are Lianghua Huang, Zhi-Fan Wu, Wei Wang, Yupeng Shi, and Mengyang Feng.
- A.1 Core Contributors: The remaining listed core contributors are Junjie He, Chen-Wei Xie, Yu Liu, and Jingren Zhou.
A.2 Contributors
The paper lists 16 contributors alphabetically by first name.
- A.2 Contributors: The contributors are Ang Wang, Bang Zhang, Baole Ai, Chen Liang, Cheng Yu, Chongyang Zhong, Jinwei Qi, Kai Zhu, Pandeng Li, Peng Zhang, Wenyuan Zhang, Xinhua Cheng, Yitong Huang, Yun Zheng, Yuzheng Wang, and Zoubin Bi.The ordering is alphabetical by first name.