Source-linked AI summary
Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation
Chunyu Li, Jiaye Li, Ruiqiao Mei, Haoyuan Xia, Hao Zhu, Jingdong Wang, Siyu Zhu
TL;DR
Real-time text-driven joint audio-video avatar generation must produce synchronized speech and portrait video, but existing models can be too slow for interaction and aggressive distillation can degrade quality. Hallo-Live combines asynchronous dual-stream diffusion with human-centric preference-guided distillation. It reports 20.38 FPS and 0.94-second latency on two H200 GPUs, with substantially higher throughput and lower latency than Ovi while maintaining strong generation quality.
Problem
Real-time joint audio-video avatar generation faces models that are too slow for interactive use and distillation that can degrade visual fidelity, speech quality, and cross-modal consistency.
Source: §1 · p. 2, §3.1 · p. 3Method
Hallo-Live combines asynchronous dual-stream diffusion, which gives video access to synchronous and short-horizon future audio, with HP-DMD reward-based reweighting for synchronization, visual fidelity, and speech naturalness.
Source: §1 · p. 2Results
20.38 FPS and 0.94 seconds latency on two NVIDIA H200 GPUs deliver 16.0× higher throughput and 99.3× lower latency than Ovi while maintaining strong VideoAlign, Sync, and human-centric fidelity results.
Source: §5 · p. 9Takeaways & Limitations
The authors describe Hallo-Live as a practical step toward deployable interactive avatar generation, with reported generalization across photorealistic, multi-speaker, and stylized prompts.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Real-time text-driven joint audio-video avatar generation requires jointly synthesizing portrait video and speech with high fidelity and precise synchronization, yet existing audio-visual diffusion models remain too slow for interactive use and often degrade noticeably after aggressive acceleration. We present Hallo-Live, a streaming framework for joint audio-visual avatar generation that combines asynchronous dual-stream diffusion with human-centric preference-guided distillation. To reduce articulation lag in causal generation, we introduce Future-Expanding Attention, which allows each video block to access synchronous audio together with a short horizon of future phonetic cues. To mitigate the quality loss of few-step distillation, we further propose Human-Centric Preference-Guided DMD (HP-DMD), which reweights training samples using rewards from visual fidelity, speech naturalness, and audio-visual synchronization. On two NVIDIA H200 GPUs, Hallo-Live runs at 20.38 FPS with 0.94 seconds latency, yielding 16.0x higher throughput and 99.3x lower latency than the teacher model Ovi. Despite this speedup, it retains strong generation quality, reaching comparable VideoAlign overall score and Sync Confidence score while outperforming other accelerated baselines in the overall quality-efficiency trade-off. Qualitative results further show robust generalization across photorealistic, multi-speaker, and stylized scenarios. To the best of our knowledge, Hallo-Live is the first framework to combine streaming dual-stream diffusion with preference-guided distillation for real-time, text-driven audio-visual generation.
1 Introduction
Hallo-Live targets the difficulty of real-time joint audio-video generation by pairing streaming diffusion with preference-guided distillation. It reports substantially faster generation than Ovi while preserving strong quality and generalizing across diverse scenarios.
1 Introduction
- Joint audio-video avatar generation synthesizes coherent avatar video and speech from natural-language prompts.Ovi is cited as showing that dual-stream diffusion can produce high-quality synchronized audio-visual outputs.Source: §1 · p. 2
- Real-time joint audio-video avatar generation remains difficult because early diffusion models are too slow for interactive use and aggressive distillation can degrade quality.The reported degradation includes visual fidelity, speech quality, and cross-modal consistency.Source: §1 · p. 2
- Future-Expanding Attention lets video blocks use synchronous audio and a short look-ahead region for anticipatory lip motion during causal inference.Extra future audio noise is concatenated to the current audio input so the audio stream denoises a short future span.Source: §1 · p. 2
- HP-DMD reweights distillation updates using SyncNet, VideoAlign, and AudioBox rewards to favor synchronized, visually faithful, natural-sounding samples.The paper reports a more favorable quality-efficiency trade-off than vanilla DMD.Source: §1 · p. 2
- 20.38 FPS and 0.94-second latency on two NVIDIA H200 GPUs yield 16.0× higher throughput and 99.3× lower latency than Ovi.Hallo-Live maintains synchronization and visual alignment comparable to the teacher while surpassing previous accelerated baselines in overall quality-efficiency trade-off.Source: §1 · p. 2
- Qualitative results show robust generalization across photorealistic, multi-speaker, and stylized scenarios.Source: §1 · p. 2
2 Related Work
Prior work spans audio-driven portrait animation, joint audio-video generation, and diffusion distillation with preference alignment. Hallo-Live brings these directions together for text-driven streaming generation.
Portrait Animation and Talking Avatars
Joint Audio-Video Generation
- Joint-generation work ranges from sequential multi-modal denoising and hierarchical diffusion transformers to twin-backbone, expert-stitching, single-stream, MoE, and streaming approaches.The passage identifies MM-Diffusion [30], JavisDiT [18], Ovi [19], UniVerse-1 [40], DaVinci-MagiHuman [4], MOVA [34], LTX-2 [10], and OmniForcing [33] as examples.Source: §2 · p. 2
Distribution Matching Distillation and Preference Alignment
- Distribution-matching distillation accelerates diffusion, while preference-alignment methods steer outputs toward human aesthetics; newer work combines reward learning with distribution matching.Source: §2 · p. 2
Related-work positioning
- Hallo-Live positions its preference-aware reweighting for the dual-modal demands of visual fidelity, speech naturalness, and timestamp-level synchronization.The approach is presented as bridging streaming dual-stream diffusion and preference-guided distillation.Source: §2 · pp. 2–3
- Figure 2 depicts Stage I adapting the pretrained dual-stream model and Stage II performing autoregressive self-rollout with an audio-video KV cache and reward-weighted dual-stream DMD.The fusion blocks combine single-modal block-causal self-attention, text cross-attention, and cross-modal attention.Source: Figure 2 · p. 3
3 Method
Hallo-Live adapts joint audio-video diffusion for streaming through asynchronous audio look-ahead and preference-guided distillation. Its training pipeline first transfers a pretrained teacher’s denoising behavior, then uses autoregressive self-rollout and modality-specific DMD.
3.1 Overview
- Streaming adaptation faces two bottlenecks: unavailable near-future phonetics for articulation and vanilla DMD’s degradation of human-centric quality.The stated degradation includes loss of fine facial and body textures, robotic prosody, and temporal drift in audio-video alignment.Source: §3.1 · p. 3
3.2 Asynchronous Dual-Stream Diffusion
- Future-Expanding Attention keeps video focused on its current block while audio provides historical, synchronous, and provisional look-ahead context.The provisional future audio block is refreshed rather than committed, providing anticipatory cues with one-block look-ahead latency.Source: §3.2 · pp. 3–4
- The Future-Expanding Block-Causal Mask exposes the next aligned audio tokens to video queries while keeping later future audio inaccessible.This mask matches training visibility to streaming inference; setting W=0 recovers strict block-causal visibility.Source: §3.2 · p. 4
3.3 Human-Centric Preference-Guided DMD
- HP-DMD reweights samples using visual-fidelity, acoustic-naturalness, and synchronization rewards, tilting distillation toward higher-reward teacher regions.The rewards are standardized within each batch before aggregation into sample weights.Source: §3.3 · pp. 4–5
3.4 Architecture and Training Pipeline
- Stage I adapts a pretrained Ovi backbone to causal and Future-Expanding masks, regressing the student toward the teacher trajectory before autoregressive rollout.This initialization transfers the teacher’s joint audio-video denoising capabilities to the student.Source: §3.4 · p. 5
- Stage II rolls out audio-video blocks autoregressively with an online KV cache and applies separate DMD gradients to video and audio.Decoded-rollout rewards modulate the video DMD term with VideoAlign and SyncNet, and the audio term with AudioBox and SyncNet.Source: §3.4 · p. 5
4 Experiments
Experiments compare Hallo-Live with joint audio-video systems, evaluate efficiency and generation quality, and ablate attention and preference rewards. Results show real-time generation with quality close to Ovi, while the ablations examine future audio context and reward balance.
4 Experiments
- Training uses 3,000 Stage I steps and 2,000 Stage II steps; the data pipeline expands 100 human-written seed prompts and filters prompts for stable synchronization and reliable speech.The expanded prompts use Qwen3.5-Plus rewriting and paraphrasing before Ovi generates audio-video samples.Source: §4 · p. 6
- Evaluation covers efficiency, video fidelity, synchronization, acoustic naturalness, text-speech consistency, and portrait fidelity, with throughput and latency measured on two H200 GPUs.VideoAlign, SyncNet, AudioBox, TTS-oriented metrics, and VBench-2.0 portrait-fidelity measures provide the reported quality assessments.Source: §4 · pp. 6–7
4.1 Comparison results
- 20.38 FPS and 0.94 seconds latency deliver 16.0× higher throughput and 99.3× lower latency than Ovi, while VideoAlign Overall is 2.32 versus Ovi’s 2.40.Hallo-Live also achieves a Sync score of 4.72 and the strongest quality-efficiency trade-off among the compared systems.Source: Table 1 · p. 7, §4.1 · pp. 7–8
4.2 Ablation results
- Individual rewards primarily improve their supervised modality, while all three together yield 2.32 VideoAlign and 4.72 Sync, with sharper details and more accurate lip-audio alignment than vanilla DMD.The joint setting is more balanced across visual fidelity, synchronization, and acoustic naturalness than single-reward optimization.Source: Table 3 · p. 8, §4.2 · p. 8, Figure 8 · p. 9
- Sync Confidence score rises from 3.87 to 4.08, 4.22, 4.29, and 4.33 as W grows from 5 to 30, with marginal gains after W=15.The ablation compares strict block-causal attention with Future-Expanding Attention at different window sizes.Source: Figure 7 · p. 7, §4.2 · p. 8, Table 2
- β=2 is a universal sweet spot: in Sync-only, the Sync score rises from 4.58 at β=1 to 5.37 at β=2, then drops to 3.30 and 3.11 at β=3 and β=4.The paper attributes the deterioration above β=2 to reward hacking and compromised generation stability and quality.Source: §4.2 · pp. 8–9
5 Conclusion
Hallo-Live combines asynchronous dual-stream generation with Human-Centric Preference-Guided DMD to improve lip-audio synchronization under streaming inference while reducing quality loss from aggressive acceleration. It reports 20.38 FPS and 0.94 seconds latency on two NVIDIA H200 GPUs and generalizes across diverse prompt conditions.
5 Conclusion
- Combining Asynchronous Dual-Stream and Human-Centric Preference-Guided DMD improves lip-audio synchronization under streaming inference while reducing quality loss from aggressive acceleration.Source: §5 · p. 9
- 20.38 FPS with 0.94 seconds latency on two NVIDIA H200 GPUs, with 16.0× higher throughput and 99.3× lower latency than the Ovi teacher.The conclusion also reports strong VideoAlign, Sync, and human-centric fidelity results.Source: §5 · p. 9
- Hallo-Live generalizes across photorealistic portraits, multi-speaker scenes, and stylized avatars.The conclusion describes these results as a practical step toward deployable interactive avatar generation.Source: §5 · p. 9
A Additional Implementation Details
Hallo-Live streams audio-video blocks using provisional future audio context and a rolling KV cache, then overwrites the provisional audio when it becomes current. Stage II training freezes the video stream after joint training so audio can continue converging.
A Additional Implementation Details
- The video branch attends to the current visual block while the audio branch uses previous, current, and provisional next-block audio context.Clean committed features enter the rolling KV cache as the temporal window advances.Source: A Additional Implementation Details · p. 11
- The provisional future audio block is regenerated when it becomes current rather than committed, preventing speculative audio errors from accumulating.This lets video use short-horizon future phonetic cues.Source: A Additional Implementation Details · p. 11
- Stage II trains both streams jointly for 2,000 steps, then freezes video and updates audio for another 1,500–2,500 steps.Video typically stabilizes around 2,000 steps, while audio usually needs 3,500–4,500 Stage II steps; stopping before audio convergence is associated with WER around 0.2–0.3.Source: A Additional Implementation Details · p. 11
B Data Pipeline
The data pipeline expands and deduplicates prompts, synthesizes paired audio-video samples with Ovi, and filters samples using multimodal quality diagnostics. The resulting corpus contains 20,000 prompts and approximately 28 hours of paired training data.
B Data Pipeline
- 100 human-written seed prompts are expanded with Qwen3.5-Plus into an initial pool of 200,000 candidate prompts.Source: B Data Pipeline · p. 11
- Cosine similarity deduplication at a 0.95 threshold reduces the pool to 30,000 distinct prompts while preserving semantic intent and diversity.Source: B Data Pipeline · p. 11
- Ovi synthesizes approximately 42 hours of paired samples for the retained prompts, then multimodal quality filtering yields 20,000 prompts and approximately 28 hours of training data.Retention requires zero WER, VideoAlign VQ of at least -0.8, VideoAlign TA of at least 0.8, Sync confidence of at least 3.0, and VBench human anatomy score of at least 0.7.Source: B Data Pipeline · p. 11