PaperlayerView exact revision on arXiv

Source-linked AI summary

X-Streamer: Unified Human World Modeling with Audiovisual Interaction

You Xie, Tianpei Gu, Zenan Li, Chenxu Zhang, Guoxian Song, Xiaochen Zhao, Chao Liang, Jianwen Jiang, Hongyi Xu, Linjie Luo

arXiv:2509.21574v1cs.CV

TL;DR

X-Streamer addresses the challenges of synchronizing streaming text, speech, and video, maintaining long-range audiovisual consistency, and generating responses in real time. Its Thinker–Actor architecture unifies multimodal understanding and generation; the framework supports real-time open-ended audiovisual interaction on two A100 GPUs while preserving long-range conversational coherence, identity, and expressive alignment.

  • Problem

    Streaming multimodal generation must synchronize text, speech, and video while maintaining long-range audiovisual consistency and real-time efficiency.

    Source: §1 · p. 2

  • Method

    X-Streamer uses a Thinker–Actor dual-transformer architecture to unify multimodal understanding and generate synchronized streaming text, speech, and video responses.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    Two A100 GPUs support real-time, open-ended multimodal interaction with long-range conversational coherence, characteristic identity, and expressive alignment across speech and motion.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    X-Streamer is a step toward persistent, interactive, and intelligent digital humans and world modeling.

    Source: §5 · p. 9

    Training solely on real-human talking-head videos limits generalization to broader scenarios.

    Source: §5 · p. 9

Abstract

from arXiv · show

We introduce X-Streamer, an end-to-end multimodal human world modeling framework for building digital human agents capable of infinite interactions across text, speech, and video within a single unified architecture. Starting from a single portrait, X-Streamer enables real-time, open-ended video calls driven by streaming multimodal inputs. At its core is a Thinker-Actor dual-transformer architecture that unifies multimodal understanding and generation, turning a static portrait into persistent and intelligent audiovisual interactions. The Thinker module perceives and reasons over streaming user inputs, while its hidden states are translated by the Actor into synchronized multimodal streams in real time. Concretely, the Thinker leverages a pretrained large language-speech model, while the Actor employs a chunk-wise autoregressive diffusion model that cross-attends to the Thinker's hidden states to produce time-aligned multimodal responses with interleaved discrete text and audio tokens and continuous video latents. To ensure long-horizon stability, we design inter- and intra-chunk attentions with time-aligned multimodal positional embeddings for fine-grained cross-modality alignment and context retention, further reinforced by chunk-wise diffusion forcing and global identity referencing. X-Streamer runs in real time on two A100 GPUs, sustaining hours-long consistent video chat experiences from arbitrary portraits and paving the way toward unified world modeling of interactive digital humans.

1 INTRODUCTION

Digital human agents still lack open-ended, long-range interaction that unifies text, speech, and video. X-Streamer addresses this with a unified Thinker–Actor model and reports coherent, real-time audiovisual streams on two A100 GPUs.

1 INTRODUCTION

  • Digital human agents need infinite streaming multimodal interaction with long-range multi-turn context and context-aware audiovisual responses.The introduction frames these capabilities as a new generative paradigm for conversational interactions at head-portrait scale.Source: §1 · p. 2
  • Modular systems face unidirectional context flow, latency, handcrafted alignment, and consistency challenges; unified text–speech and text–image work leaves text–speech–video generation largely unexplored.Long-form interactions intensify identity, motion, and context consistency challenges under compute and memory constraints.Source: §1 · p. 2
  • The framework jointly understands and generates text, speech, and video from a portrait and streaming text or audio queries.It is trained end-to-end on unlabeled human talking videos and targets synchronized, context-aware responses across extended conversations.Source: §1 · p. 2
  • A pretrained language–speech Thinker interprets user inputs, and an autoregressive Actor turns its hidden states into interleaved text, audio, and video outputs.Chunk-wise diffusion, cross-attention, synchronized positional embeddings, and global image conditioning support alignment and long-horizon stability.Source: §1 · p. 2
  • X-Streamer produces infinite audiovisual streams with long-range conversational coherence, characteristic identity, and expressive speech–motion alignment on two A100 GPUs.The model has 18B parameters and was trained on 4,248.6 hours of talking-head videos.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work addresses video streaming and real-time audiovisual interaction through separate approaches, but unified understanding and generation across text, speech, and video remains the target of X-Streamer.

Autoregressive Video Diffusion

  • Fixed-length diffusion limits variable-horizon streaming, sliding-window methods face restricted-context discontinuities, and autoregressive generation can drift over long horizons.Asynchronous diffusion methods use independent frame noise schedules to reduce drift and corruption across extended sequences.Source: §2 · p. 3

Real-Time AudioVisual Interaction

  • Real-time language–speech models provide low-latency spoken interaction, while many audiovisual systems pair speech generation with a talking-head renderer.Portrait-animation methods can improve expressiveness or lip synchronization, but the described approaches lack multi-turn memory and semantic reasoning or rely on externally supplied text.Source: §2 · pp. 3–4
  • Joint speech-and-video generation has emerged, but Veo3 and OmniTalker still rely on externally provided text inputs for content.X-Streamer instead unifies multimodal understanding and generation for context-aware audiovisual responses.Source: §2 · p. 4

3 METHOD

X-Streamer treats portrait-based, multi-turn audiovisual dialogue as unified multimodal understanding and generation. Its Thinker–Actor design synchronizes streaming text, speech, and video, with chunk-wise mechanisms for long-range video coherence and real-time inference.

3.1 UNIFIED HUMAN WORLD MODELING

  • A frozen pretrained Thinker interprets streaming text and audio, while an Actor translates its hidden states into synchronized multimodal outputs in two-second chunks.The Actor uses modality-specific generators, including a parallel video transformer initialized from pretrained language-model weights.Source: §3.1 · p. 5
  • Text and speech tokens interleave with video chunks so video generation follows text–audio semantics without waiting for the full speech output.The sequence alternates 13 text tokens and 26 speech tokens; each video segment represents about 2.08 seconds at 25 fps.Source: §3.1 · p. 5
  • Video tokens cross-attend to aligned Thinker states, while spatial self-attention within chunks and causal attention across chunks preserve audiovisual alignment and video history.Time-aligned positional embeddings are assigned to the text–audio states and video tokens.Source: §3.1 · p. 5, Figure 3 · p. 6

3.2 REAL-TIME STREAMING VIDEO GENERATION

  • Chunk-wise diffusion forcing trains video chunks with independent noise levels under noisy historical context to reduce drift from imperfect histories.The approach addresses teacher-forcing mismatch, which can cause irreversible drift and frame corruption when continuous video latents are generated autoregressively.Source: §3.2 · p. 6
  • Each video latent can attend to the portrait as a global condition while also drawing on visual history, supporting long-range identity consistency without a heavyweight reference network.The model is described as balancing identity cues from the global embedding with historical context.Source: §3.2 · p. 6
  • A standard KV cache avoids redundant computation during autoregressive generation, while the scheduler reduces denoising from |c|×N forward passes for |c| chunks and N steps to |c| + N −1.Visual context is restricted to 2K tokens (a 10-second window), while conversational context remains unchanged at up to 8K tokens (roughly 10 minutes of dialogue). The Thinker and Actor are hosted separately across two A100 GPUs.Source: §3.2 · pp. 6–7

4 EXPERIMENTS

Experiments evaluate X-Streamer’s data, training, and real-time inference setup, compare it with portrait-animation and video-generation baselines, and ablate components linked to long-horizon stability. The reported results span visual fidelity, identity, synchronization, motion, and qualitative long-duration consistency.

4.1 IMPLEMENTATION DETAILS

  • The dataset comprises about 2.7 million talking-head clips totaling 4,248.6 hours, processed as text, speech, and video triplets.The clips combine public datasets and licensed online sources.Source: §4.1 · p. 7

4.2 EVALUATION

  • Evaluation uses 50 in-the-wild reference images and 50 randomly generated multi-turn queries, comparing portrait-animation and video-generation baselines under different conditions.Portrait-animation baselines receive audio synthesized by X-Streamer; video-generation baselines are text-prompted, and some require hours to synthesize a one-minute video.Source: §4.1 · p. 7, §4.2 · p. 8

4.1 IMPLEMENTATION DETAILS

  • Training uses a pretrained GLM-4-Voice Thinker while training the Actor’s video transformer; inference assigns text and speech generation to GLM-4-Voice and synchronized multimodal translation to the Actor.Training has pretraining on 2.7 million clips and finetuning on 220,000 high-quality long-form samples; the full model is not instruction-finetuned.Source: §4.1 · pp. 7–8

4.2 EVALUATION

  • X-Streamer outperforms all baselines in CPBD, FVD, VQ, Glo, Div, and Lip, ranks second in ID-Sim, and achieves the lowest SynD.In the user study, it leads identity preservation (ID↑); the authors attribute SadTalker’s higher objective ID similarity to its restricted motion and zoomed-in framing.Source: Table 1 · p. 8, §4.2 · p. 9

4.1 IMPLEMENTATION DETAILS

  • Distributing the Thinker and Actor across two A100 GPUs yields 25 fps multimodal streaming.A one-minute response takes 51.2 seconds for the Thinker and 58.2 seconds for the Actor; generation uses 8-frame video chunks interleaved with text and speech tokens.Source: §4.1 · p. 8

4.2 EVALUATION

  • X-Streamer remains stable with consistent identity throughout long sequences, while CausVid degrades after around 10 seconds and SkyReels-V2 and MAGI-1 show drift within 30 seconds.Qualitatively, X-Streamer also handles chest-level portraits, occlusions, side views, and complex environments, with more coordinated head movements and expressive gestures than the compared portrait-animation methods.Source: §4.2 · p. 8
  • The evaluation covers visual fidelity, identity, audiovisual synchronization, motion dynamics, and user judgments of identity, lip synchronization, motion diversity, and video quality.The user study includes 20 participants and 100 choices per dimension.Source: §4.2 · pp. 8–9

4.3 ABLATION STUDY

  • Replacing diffusion forcing, global identity referencing, or spatially bidirectional within-chunk attention with alternatives produces motion drift, identity and color drift, or weaker temporal coherence and visual fidelity.The visual ablation reports reduced flickering and better structural integrity with spatially bidirectional attention than with fully causal token-wise attention.Source: §4.3 · p. 9, Figure 5 · p. 9

5 CONCLUSION

X-Streamer unifies text, speech, and video understanding and generation through a Thinker–Actor dual-transformer design. The authors position it as a step toward persistent, interactive digital humans, while noting that training on real-human talking-head videos limits generalization to broader scenarios.

5 CONCLUSION

  • X-Streamer’s Thinker reasons conversationally, while its Actor translates Thinker hidden states into synchronized streaming multimodal responses.Chunk-wise diffusion forcing is used to balance real-time efficiency, long-range consistency, and temporal synchronization.Source: §5 · p. 9
  • The authors describe X-Streamer as a step toward persistent, interactive, intelligent digital humans and world modeling.They propose expanding beyond conversational interactions to user-video perception, object interaction, and multimodal commands.Source: §5 · pp. 9–10
  • The authors identify training solely on real-human talking-head videos as limiting generalization to broader scenarios.They also leave higher-resolution real-time video with ultra-long audiovisual context and broader multimodal engagement for future work.Source: §5 · pp. 9–10

A.1 DATASET

The dataset is a large-scale audiovisual talking-head corpus, filtered for alignment and visual quality and processed through a sequence of temporal, speaker, and modality checks.

A.1 DATASET

  • The corpus contains 2,780,920 bilingual English–Chinese samples totaling 4,248.6 hours; other languages are excluded because the GLM-4-Voice speech tokenizer does not support them.Its supervised fine-tuning subset contains 217,074 samples totaling 331.64 hours.Source: §A.1 · p. 16
  • Longer fine-tuning samples are selected using quality thresholds, including IQA of at least 70, exactly one detected speaker, no OCR-detected overlays, and SyncNet of at least 5.0.The subset includes 5,406 samples longer than 20 seconds, with an average length of 41 seconds per sample.Source: §A.1 · p. 16
  • The curation pipeline segments scenes, filters lip-sync quality, checks for one visible speaker, tracks faces, removes low-quality shots and overlays, and denoises audio.It then transcribes speech and caches text, audio, and video embeddings to reduce training-time I/O and preprocessing overhead.Source: §A.1 · pp. 16–17

A.2 DIFFUSION FORCING DENOISING SCHEDULING

The chunk-wise pyramid scheduler denoises chunks sequentially while allowing chunk-level parallelism during inference. It reduces the required forward passes from |c| × N to |c| + N −1.

A.2 DIFFUSION FORCING DENOISING SCHEDULING

  • |c| + N −1 forward passes replace the |c| × N required by chunk-by-chunk DDIM scheduling.The scheduler is illustrated in Figure 6.Source: Figure 6 · p. 17, §A.2 · p. 17
  • Kchunk applies diffusion-forcing denoising sequentially across chunks, enabling chunk-level parallelism during inference.Source: §A.2 · p. 17
  • Each matrix row assigns token noise levels for a denoising round, and the matrix height gives the total forward-pass count.Denoising starts from a fully noised sequence and progressively refines chunks in order.Source: §A.2 · p. 17

A.3 EXTENDING X-STREAMER WITH VISUAL PERCEPTION

The visual-perception extension adds an auxiliary VLM to supply live webcam context to X-Streamer. The current system otherwise accepts text and speech queries because GLM-4-Voice does not process visual inputs.

A.3 EXTENDING X-STREAMER WITH VISUAL PERCEPTION

  • Because GLM-4-Voice does not process visual inputs, the current system is limited to text and speech queries.Source: §A.3 · p. 17
  • An auxiliary VLM analyzes the latest webcam frame and passes a concise scene description into the Thinker’s prompt.Source: §A.3 · p. 17
  • The added visual context lets the model reason over user queries and current scene information without manual annotations.Source: §A.3 · p. 17

A.4 MORE RESULTS

The supplied captions identify Figure 7 as a visual-perception example and Figure 8 as additional X-Streamer results, without specifying their outcomes.

A.4 MORE RESULTS

  • Figure 7 depicts a VLM analyzing a webcam frame and supplying a scene description to guide a multimodal response.Source: Figure 7 · p. 18
  • Figure 8 is captioned as additional X-Streamer results, with no further result details in the caption.Source: Figure 8 · p. 19

Related papers

Loading 2509.21574v1…