Source-linked AI summary
StreamChar: Long-Horizon Streaming Character Audio-Video Generation with Decoupled Orchestration
Linrui Tian, Qi Wang, Bang Zhang
TL;DR
Real-time streaming character audio-video generation must balance transcript fidelity, visual continuity, and low latency. StreamChar decouples orchestration from local denoising and achieves real-time long-horizon generation on a single GPU while remaining competitive across key quality and stability measures.
Problem
Real-time streaming joint audio-video character generation remains challenging because autoregressive error accumulation and few-step distillation create competing demands for quality and stability.
Method
StreamChar decouples long-horizon orchestration from short-window audio-video denoising, using two-stage distillation with transcript alignment and persistent sink-frame conditioning.
Results
StreamChar supports long-horizon real-time streaming audio-video generation on a single GPU while remaining competitive with recent streaming and non-streaming methods.
Takeaways & Limitations
StreamChar offers a competitive balance of transcript fidelity, audio-visual synchronization, visual quality, and streaming stability within its evaluated setting.
Takeaways & Limitations
Real-time performance is measured only on a single H100 GPU with a 33-frame chunk budget, so lower-end deployment may require further optimization.
Abstract
from arXiv · showhide
Real-time streaming joint audio-video generation for character animation requires a generator to speak the requested transcript, maintain visual identity across chunks, and run within a strict playback budget. These requirements are difficult to satisfy simultaneously: chunk-wise autoregressive generation can accumulate transcript-audio misalignment and visual drift, while the few-step distillation needed for low latency often degrades spatial diversity and temporal quality. We present StreamChar, a streaming framework that separates long-horizon orchestration from short-window audio-video denoising. An LLM-based orchestrator uses the transcript and historical context to produce frame-aligned audio conditions, and a joint audio-video DiT performs local bidirectional denoising with reference and motion-frame conditioning. For efficient deployment, we use a two-stage distillation pipeline that first compresses the sampler and then fine-tunes the student under online chunk rollouts. A progress-aware pointer aligns partial transcripts with generated audio during rollout training, and a sink-chunk memory provides a persistent visual anchor for reducing long-horizon drift. Experiments on short-clip and long-horizon protocols show that StreamChar runs in real time on a single H100 GPU and provides a favorable system-level trade-off among transcript fidelity, audio-visual synchronization, visual quality, and streaming stability compared with recent joint and audio-driven baselines.
1. Introduction
StreamChar addresses the coupled challenges of transcript-faithful long-horizon coherence and real-time inference in streaming character audio-video generation. It decouples global orchestration from local denoising and separates distillation into step compression and rollout consistency training.
- Challenges: Streaming character audio-video generation must preserve transcript–audio correspondence across chunks while generating each chunk faster than its playback duration.Chunk-wise decoding can cause omissions, repetitions, and semantic misalignment, while diffusion inference typically requires tens to hundreds of denoising steps.
- Challenges: Aggressive distillation can cause mode collapse and reduced spatial diversity, while autoregressive deployment accumulates errors that progressively degrade video quality.The introduction identifies distillation-induced stereotyped spatial behaviors and progressive video drifting as coupled failure modes.
- Architecture: StreamChar uses a decoupled LLM orchestrator and short-window DiT, assigning transcript and historical-context planning to the orchestrator and joint local audio-video denoising to the DiT.Motion-frame conditioning maintains cross-chunk continuity while the denoising backbone uses full bidirectional attention within short windows.
- Stability mechanisms: A progress-aware pointer aligns partial transcripts with generated audio, and a sink-frame mechanism provides a persistent visual anchor that suppresses long-horizon video drift.The first chunk serves as the sink-frame anchor attended by subsequent chunks during long rollouts.
- Training and evaluation: StreamChar separates step compression from rollout consistency training in a two-stage distillation recipe and supports real-time long-horizon streaming on a single GPU while remaining competitive with recent methods.Stage I performs step compression, whereas Stage II trains rollout consistency with transcript alignment and sink-frame mechanisms.
2. Related Work
Prior work establishes diffusion and DiT models as the backbone of audio-video generation, while highlighting limitations in long-form streaming, multimodal coordination, and efficient chunk-wise deployment. StreamChar addresses these gaps by combining LLM-based orchestration with short-window bidirectional DiT denoising.
- Diffusion models for audio-video generation: Monolithic DiTs unify text, audio, and visual tokens effectively for short clips but face challenges when scaled to long-form streaming.The shared backbone must simultaneously handle semantic und
- LLMs as planners and conditioners: LLMs are increasingly used as planners that decompose complex visual prompts into structured layouts or storyboards guiding downstream diffusion models.In audio, recent works also leverage LLMs, as indicated by the supplied passage.
- Audio-driven video generation: Early audio-driven video systems provide high-fidelity portrait animation but require dozens of denoising steps, whereas recent streaming methods achieve sub-second latency through knowledge distillation.These systems provide strong references for video quality and latency but do not address joint text-to-audio-video generation.
- Audio-driven video generation: Audio-driven systems simplify synchronization but do not jointly generate speech content and visual motion, motivating explicit cross-modal coordination in StreamChar.StreamChar uses decoupled LLM orchestration and short-window bidirectional DiT denoising for this joint setting.
- Knowledge distillation for efficient diffusion: Progressive distillation, consistency models, and DMD compress multi-step diffusion samplers into few-step generators, but chunk-wise streaming can encounter distillation-induced mode collapse.The supplied passage identifies distillation-induced mode collapse as one of two intertwined failure modes, with the second failure mode truncated.
3. Method
StreamChar separates long-horizon transcript and history planning from local joint audio-video denoising. A causal orchestrator produces frame-aligned continuous audio conditions, while a conditioned DiT synthesizes coherent audio and video through cross-modal attention and persistent visual controls.
- DiT: The joint DiT denoises audio and video latents conditioned on prompt semantics, the orchestrator output, reference frames, and previously generated motion frames.Reference frames anchor identity and temporal consistency, while motion frames improve cross-chunk coherence.
- Orchestrator: The orchestrator reads the transcript, reference inputs, and long-term generated history to produce a frame-aligned continuous audio condition for each DiT denoising step.It is coupled to the noisy audio latent and shared timestep, but does not perform diffusion itself.
- Continuous conditioning: The orchestrator forms its continuous condition without an audio tokenizer by concatenating reference-audio, transcript, history-audio, and current-step conditioning embeddings.The current-step tail contains the noisy audio latent and timestep, and its final-layer hidden states produce ca jointly with the DiT through the diffusion flow loss.
- Audio alignment and joint attention: Shared self-attention lets noisy audio and video tokens exchange lip-motion, prosody, and scene-dynamics information directly at the token level.A modality-aware two-expert MoE then routes audio and video tokens to distinct feed-forward experts while retaining shared attention projections.
- Conditioning and temporal alignment: Clean-state condition embeddings and asymmetric masking make reference and motion controls timestep-invariant while restricting attention to a unidirectional flow from noisy latents to condition tokens.Modality-aware RoPE also aligns audio and video tokens across their 4:1 latent-rate difference and preserves temporal continuity across chunks with offset-aware indexing.
4. Streaming Inference and Distillation
StreamChar preserves bidirectional denoising within each streaming chunk while using historical motion conditioning for continuity across chunks. A two-stage distillation pipeline then combines four-step generation with online rollout consistency, transcript-progress alignment, and sink-frame memory to stabilize long-horizon inference.
- Streaming denoising: Bidirectional attention spans all noisy audio-video tokens within the current window, while historical motion conditioning maintains continuity across chunks.This avoids the quality loss of purely causal denoising without requiring architectural causality across the full sequence.
- Streaming denoising: 9 latent video frames per chunk decode to 33 RGB frames, yielding approximately 1.38 s theoretical latency at 24 fps before codec and orchestration overhead.The student rolls forward using motion latents from prior decoded output while restricting bidirectional attention to the current chunk.
- Distillation: Two-stage distillation first compresses the pretrained 50-step sampler into a four-step generator, then trains online rollouts for autoregressive consistency.The first stage preserves single-chunk quality and reaches approximately 24 fps clip-level generation on one H100 GPU; the second uses student-generated motion latents.
- Alignment: The progress-aware pointer predicts each chunk’s spoken transcript endpoint from transcript states and audio conditions, enabling accurate transcript truncation during rollout distillation.It refines framewise soft positions with learnable offsets and confidence weights, using ASR-derived endpoints with smooth ℓ1 supervision.
- Long-horizon stability: The first student-generated chunk serves as persistently attended sink memory, providing long-range temporal reference that reduces video drift during extended rollouts.After multiple chunks, the last several are concatenated for the real-score and fake-score branches used to compute the DMD loss.
5. Experiments
Experiments show that StreamChar achieves real-time single-GPU streaming while preserving transcript alignment, visual quality, temporal continuity, and motion richness. Ablations further identify sink-chunk memory, multi-chunk rollout training, and two-stage distillation as important for stable long-horizon generation.
- Inference Efficiency: 0.96 s per 33-frame chunk is achieved on a single H100 GPU using 512 × 512 resolution and 4-step bfloat16 generation.The pipeline additionally uses approximately 0.30 s for VAE decoding, 0.05 s for preprocessing, and 0.025 s for stream writing, while reusing motion latents and overlapping preprocessing with decoding.
- Results and Analysis: 3.54% WER for the base model and 3.65% after distillation demonstrate strong speech intelligibility and transcript alignment during multi-chunk streaming.The near-identical distilled performance supports the role of the LLM orchestrator in preserving fine-grained phonetic alignment.
- Results and Analysis: 17.99 FID is the lowest among joint audio-video baselines, while StreamChar also achieves a leading Human Anatomy score despite using a 5B-parameter backbone.Its perceptual quality is reported as comparable to generators typically exceeding 14B parameters.
- Sink Chunk for Error Accumulation and Mode Collapse: Removing the sink chunk increases Quality Drift from 0.0067 to 0.0304, causing color shifts, appearance degradation, stereotyped spatial behavior, and persistent spatial offsets.The result indicates that sink conditioning reduces long-horizon error accumulation and mode collapse.
- Single-chunk vs. Multi-chunk in Stage II: 35.4% WER results from isolated one-chunk Stage II training, which also reduces Sync-C/D scores compared with multi-chunk online rollout training.The passage attributes the degradation to insufficient cross-chunk acoustic context for transcript alignment and long-range phonetic transitions.
- Two-stage vs. single-stage distillation: Two-stage distillation preserves efficiency and visual dynamics by separating few-step mapping stabilization in Stage I from cross-chunk consistency refinement in Stage II.Skipping Stage I produces motion suppression and reference-frame anchoring despite competitive short-clip metrics.
6. Limitations
StreamChar’s evaluation and deployment claims have two important limitations: audio-driven baselines use generated audio as a shared driver, and real-time performance is demonstrated only on a single H100 GPU with a 33-frame chunk budget.
- Evaluation scope: Audio-driven streaming baselines use StreamChar’s generated audio as a common driver, making them controlled video references rather than full text-to-audio-video competitors.This evaluation setup limits direct comparison with complete text-to-audio-video systems.
- Deployment constraints: Real-time performance is measured on a single H100 GPU with a 33-frame chunk budget.Deployment on lower-end hardware may require additional optimization.
7. Conclusion
StreamChar is a decoupled LLM–DiT framework for long-horizon streaming audio–video generation. Its orchestrator, denoiser, and efficiency mechanisms enable real-time chunk-wise generation with rollout stability and a competitive system balance.
- Framework: StreamChar decouples transcript-level planning in an LLM orchestrator from short-window joint denoising in a motion-conditioned DiT.The framework targets long-horizon streaming audio–video generation.
- Efficiency and stability: A two-stage distillation strategy, PAP, and sink-frame memory enable real-time chunk-wise generation with rollout stability.These components support efficient deployment and persistent temporal generation behavior.
- Evaluation: Experiments show that StreamChar runs in real time on a single GPU and offers a competitive balance of transcript fidelity and related system qualities.The supplied conclusion passage truncates the final list of system qualities after “transcript f”.
Ours
The section presents qualitative long-horizon comparisons, showing sink-chunk conditioning as a mechanism for reducing drift and repetitive spatial behavior. It also includes comparisons with several baselines and a user study under the GSB protocol.
- Qualitative comparisons: Figure 4 provides a long-horizon qualitative comparison.
- Long-horizon stability: Sink-chunk conditioning reduces long-horizon drift and repetitive spatial behavior.This finding is presented in Figure 5.
- Baseline comparisons: Figure 6 qualitatively compares StreamChar with SoulX-FlashTalk, SoulX-FlashHead, and LiveAvatar.
- User study: Figure 7 reports a user study under the GSB protocol.