Source-linked AI summary
Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models
Lianghua Huang, Zhi-Fan Wu, Wei Wang, Yupeng Shi, Mengyang Feng, Junjie He, Chen-Wei Xie, Yu Liu, Jingren Zhou, Ang Wang, Bang Zhang, Baole Ai, Chen Liang, Cheng Yu, Chongyang Zhong, Jinwei Qi, Kai Zhu, Pandeng Li, Peng Zhang, Wenyuan Zhang, Xinhua Cheng, Yitong Huang, Yun Zheng, Yuzheng Wang, Zoubin Bi
TL;DR
Real-time full-duplex audio-visual interaction must causally align modalities with different rates, representations, and latency constraints; Wan-Streamer addresses this with a native-streaming end-to-end model in one Transformer. It reports about 200 ms model-side response latency and about 550 ms total interaction latency with a 350 ms bidirectional network budget.
Problem
Real-time audio-visual interaction requires causal alignment across modalities with different rates, representations, and latency constraints, for applications including embodied assistants and real-time digital humans.
Source: §1 · pp. 1–2Method
Wan-Streamer models language, audio, and video inputs and outputs as an interleaved causal sequence in one native-streaming Transformer without external generation modules.
Source: §1 · pp. 2–3Results
About 200 ms model-side latency and about 550 ms total interaction latency with a 350 ms bidirectional network budget were measured for Wan-Streamer.
Source: §3 · p. 5Takeaways & Limitations
The paper concludes that Wan-Streamer reaches sub-second interactive latency while preserving full-history context for multimodal interaction.
Source: §5 · p. 8The current v0.1 results are validated at a preliminary 192p output resolution.
Source: §5 · p. 8
Abstract
from arXiv · showhide
We present Wan-Streamer, a native-streaming, end-to-end interactive foundation model designed from the ground up for real-time, low-latency, full-duplex audio-visual interaction. Wan-Streamer seamlessly models language, audio, and video as both input and output within a single Transformer, where the sequence is represented as interleaved visual, audio, and text input tokens together with visual, audio, and text output tokens, coordinated by block-causal attention for incremental streaming. Unlike cascaded interactive systems that rely on separate VAD, ASR, language, TTS, audio-driven animation, or video-generation modules, Wan-Streamer does not rely on external language, speech, avatar, or video-generation modules: perception, reasoning, generation, response timing, turn management, and cross-modal synchronization are learned jointly within one unified model, reducing pipeline latency and error accumulation. To support natural audio-visual responsiveness, we redesign the entire stack around streamability, including causal encoders, causal decoders, block-causal attention, and low-latency multimodal token scheduling, enabling streaming units as short as 160 ms at 25 fps. Wan-Streamer achieves approximately 200 ms model-side response latency and approximately 550 ms total interaction latency when combined with 350 ms bidirectional network latency, supporting sub-second duplex audio-visual communication. These results position Wan-Streamer as a unified, end-to-end, multimodal interactive foundation model for low-latency streaming interaction.
1 Introduction
Wan-Streamer is a native-streaming, end-to-end model that handles language, audio, and video as inputs and outputs within one Transformer. Its thinker-performer inference pipeline reaches approximately 200 ms model-side response latency and approximately 550 ms total interaction latency with 350 ms bidirectional network latency.
Introduction
- Cascaded systems can incur module-boundary waiting and recognition or synchronization errors, while making response timing, turn management, identity preservation, and long-horizon consistency difficult to learn jointly.The paper motivates real-time interactive models for embodied assistants, digital humans, live broadcasting, interactive entertainment, and online world models [14] [22] [39].Source: §1 · pp. 1–2
- Full-duplex interaction requires ongoing perception and expression to overlap, with incoming speech and video causally shaping synchronized output while generated units enter the continuing interaction history.Different modality rates, representations, objectives, and latency constraints make streamability a modeling constraint, not merely a serving optimization.Source: §1 · p. 2
- Wan-Streamer jointly models audio-visual perception, reasoning, response planning, timing, turn-taking, and generation in one Transformer, without external speech, language, animation, or video-generation modules.Its interleaved input/output sequence uses causal audio and video VAEs, causal encoders and decoders, and block-causal attention.Source: §1 · pp. 2–3, Figure 1 · p. 2
- Approximately 200 ms model-side response latency and approximately 550 ms total interaction latency are achieved with 350 ms bidirectional network latency.The thinker-performer schedule exchanges KV-cache context and overlaps current-step perception, previous-unit decoding, and next-unit latent generation.Source: §1 · p. 3
2 Method
Wan-Streamer treats interaction as a continuous causal stream, jointly training multimodal understanding and generation before adapting the model for duplex interaction and low-latency deployment. Its inference schedule separates language/state updates from latent generation to overlap work across streaming units.
2.1 Overview
- Each response uses causal history from both user observations and prior agent outputs, predicting language tokens while jointly generating audio and video latents conditioned on shared context.Clean generated latents are appended to history for subsequent units, while causal decoders render them as external audio and video.Source: §2.1 · pp. 3–4
2.2 Data
- Training data spans multimodal understanding, single- and cross-modal generation, and end-to-end duplex interaction with text, audio, and video on both sides.The duplex data targets simultaneous multimodal perception and expression.Source: §2.2 · p. 4
2.3 Training
- Training has three stages: mixed-task pretraining, duplex end-to-end interaction training, and distillation of a guided teacher into a faster student.Rolling distillation trains the student on its own consecutive-unit history using self-forcing [17] and distribution matching [45] [46] to reduce train-test mismatch and improve long-form quality.Source: §2.3 · p. 4
2.4 Inference
- Inference splits work between a thinker for perception, language/state updates, and decoding, and a performer for audio-visual latent generation, overlapping adjacent streaming units.Model-side response latency is approximately 200 ms; real-time throughput requires performer processing plus KV-cache and latent communication to fit within a 160 ms unit.Source: §2.4 · pp. 4–5, Figure 2 · p. 5
3 Experiments
Wan-Streamer reports about 200 ms model-side and about 550 ms total audio-visual response latency, while the experiments distinguish full interaction latency from partial or component-level runtime metrics. They also describe continuous, synchronized visual behavior and full-duplex interruption and proactive-speaking capabilities.
Latency and runtime comparison
- About 200 ms model-side signal-to-signal latency and about 550 ms total interaction latency are reported for a 160 ms streaming unit at 25 FPS, including a 350 ms bidirectional network budget.The measurement runs from availability of the user unit at the thinker to decoding of the corresponding audio-video response unit for emission.Source: §3 · p. 5
- Latency comparisons are separated by measurement boundary because model-only, first-packet, endpointing, and API measurements may not represent a remote user’s perceived response delay.Some omni-modal systems accept audio or video input without returning synchronized visual agent output, so their measurements are not equivalent to a full interaction loop.Source: §3 · pp. 5–6
- Runtime comparisons distinguish interactive full-loop systems from renderers and audio-visual generators, whose FPS, first-frame, or chunk metrics may omit upstream dialogue, speech, or perception.Wan-Streamer is evaluated for both covered interaction scope and runtime, rather than treating component-level speed as end-to-end latency.Source: §3 · p. 6, Table 2 · p. 7
Naturalness
- Wan-Streamer maintains identity, gaze, posture, breathing, and subtle facial motion while idle, and produces speech-coupled nonverbal feedback while listening.Because speech and video latents are predicted from the same causal context before decoding, lip motion, facial dynamics, and prosody are synchronized natively.Source: §3 · p. 7
Interruption and proactive speaking
- Interleaved interaction training and continued audio-video input during generation enable Wan-Streamer to adapt when interrupted and initiate comments or questions about salient visual events.The model learns from examples of continuation, pauses, overlap, interruption, yielding, and resumption, supporting exchange beyond a passive question-answer loop.Source: §3 · p. 7
4 Related Works
Related work spans full-duplex speech dialogue, interactive audio-visual avatars, and emerging end-to-end audio-visual agents. Wan-Streamer is positioned as a unified native full-duplex model that handles language, audio, and video on both input and output sides.
Full-duplex spoken dialogue
Interactive digital humans and audio-visual avatars
- Interactive digital-human systems connect multimodal dialogue or understanding with avatar generation, but often rely on separate modules and system-level turn control [6] [8] [13] [28] [40] [47].Streaming avatar methods improve latency and temporal consistency, while duplex behavior often remains outside end-to-end learning [10] [21] [25] [32] [34].Source: §4 · p. 8
End-to-end audio-visual interaction
- Body of Her integrates audio and visual inputs to model speech, full-body behavior, idling, response, and manipulation in real time [1]; FlowAct-R1 and related humanoid video models point toward action-level interactive generation [38].Source: §4 · p. 8
- Wan-Streamer extends end-to-end interaction by modeling language, audio, and video on both input and output sides in one Transformer with fully causal streaming inference.This lets perception, reasoning, speaking, visible listening, interruption handling, and synchronized audio-visual generation operate as one native full-duplex process.Source: §4 · p. 8
5 Conclusion · Appendix · A Contributions and Acknowledgements
Wan-Streamer is a native-streaming, end-to-end model for real-time full-duplex text, audio, and video interaction. Its causal multimodal design enables synchronized responses with sub-second latency, with v0.1 results validated at preliminary 192p resolution.
5 Conclusion
- Wan-Streamer models user inputs and agent outputs across modalities as one causal stream in a single Transformer, rather than alternating among separate perception and generation modules.Source: §5 · p. 8
- Fully causal audio and video VAEs, causal encoders and decoders, and block-causal attention support low-delay perception and synchronized audio-visual responses.Source: §5 · p. 8
- The model emits streaming units and commits generated latents to history, while the thinker-performer serving design supports sub-second latency with full-history context.Source: §5 · p. 8
- The current v0.1 results are validated at preliminary 192p output resolution as a proof of concept for the end-to-end streaming design.Source: §5 · p. 8
- Scaling to higher resolutions is described as straightforward and left to future work.Source: §5 · p. 8
- The authors argue that real-time multimodal agents should be designed as native full-duplex systems, learning listening, seeing, speaking, and visible response jointly.Source: §5 · p. 8