LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation

• The paper targets the gap in research on interactive video diffusion conditioned simultaneously on text, images, and audio.Prior real-time distillation approaches primarily address text-to-video generation.Source: §1 · p. 2
• LiveTalk outperforms Veo3 and Sora2 in multi-round coherence and content quality while reducing response latency from minutes to real-time generation.The system integrates the distilled model with an audio language model and uses Anchor-Heavy Identity Sinks to preserve speaker identity in long-horizon streaming.Source: §1 · p. 2
• The proposed recipe combines curated multimodal conditions, converged ODE initialization, and an aggressive optimization schedule for stable on-policy distillation.The paper reports that ablations show joint quality improvements across perceptual metrics, audio-visual synchronization, and aesthetic quality.Source: §1 · p. 2
• The distilled 4-step model matches or exceeds bidirectional diffusion baselines while reducing inference cost by over 20×, reaching 24.82 FPS on a single GPU.The reported comparisons include the 1.3B OmniAvatar variant and larger baselines.Source: §1 · p. 2

2. Related Work

• LiveTalk introduces a multi-turn interaction benchmark and outperforms Veo3 and Sora2 in multi-video coherence and content quality metrics.The system also achieves sub-second response, according to the paper's contribution summary.Source: §2 · p. 3

2.1 Multimodal Video Diffusion

• Multimodal video diffusion supports image-guided editing and virtual avatars, but faces challenges in architecture, training overhead, and cross-modal alignment.Many-step diffusion across full sequences also carries inference cost and latency that limit real-time use.Source: §2.1 · p. 3

2.2 Real-Time Video Diffusion

• Real-time video diffusion combines autoregressive streaming with diffusion fidelity, while exposure bias from accumulated errors remains a challenge.Distillation and self-generated rollouts have been studied mainly for text-to-video rather than multimodal video diffusion.Source: §2.2 · p. 3

3.1 Preliminary: Self Forcing

• Self Forcing first distills an ODE trajectory into a few-step causal student, then applies on-policy DMD rollouts to mitigate exposure bias.The student generates video block by block; ODE initialization uses sampled timesteps from the teacher trajectory to predict clean latents.Source: §3.1 · p. 3

3.2 Issues on Existing Distillation Recipe

• Naive multimodal Self Forcing DMD can produce flickering, black frames, and quality degradation through unstable interactions between generator rollouts and critic training.Corrupted rollouts can yield inaccurate critic gradients that further degrade the generator; the figure contrasts degraded Self Forcing results with stable results from the proposed method.Source: §3.2 · p. 4, Figure 2 · p. 4
• Low-quality reference images can destabilize distillation, while filtering by brightness and quality metrics improves stability only slightly and does not prevent emerging blurriness.The reported examples include low overall image quality in Hallo3 and facial blurriness in HDTF.Source: §3.2 · p. 4
• Insufficient ODE initialization can cause collapse or blurry plateaus, and multimodal DMD reaches a peak before degrading within a few hundred steps.The short learning window can leave multimodal alignment, especially audio-visual synchronization, insufficiently learned.Source: §3.2 · p. 4, §3.3 · p. 5

3.3 Improvements on Existing Distillation Recipe

• The improved recipe curates image and text conditions, trains ODE initialization to convergence, and uses doubled learning rates with higher teacher CFG to learn before degradation.The authors report stronger audio-visual alignment while maintaining high visual quality, with aggressive settings framed as a trade-off that can introduce instability.Source: §3.3 · p. 5

• Qwen3-Omni generates streaming audio responses, while the video model uses audio, a reference avatar image, and motion text prompts to generate video.Generation proceeds autoregressively in blocks of three latent frames with four diffusion steps per block, using prior-block KV cache for visual consistency.Source: §4 · p. 5
• Overlapped audio windowing supplies adjacent-frame acoustic context without waiting to encode an entire audio clip before video generation.The method encodes and generates once a small windowed segment is available to support smooth lip-sync and natural motion at real-time responsiveness.Source: §4 · p. 5
• The modular system separates real-time avatar rendering from reasoning and streaming speech generation, with sub-second first-frame latency.Both modules contribute to streaming video synchronized with audio.Source: §4 · p. 5

5. Experiments

• Anchor-Heavy Identity Sinks preserve speaker appearance over several minutes, while denoising and decoding in parallel reduces per-block latency to the slower stage.The KV window uses three identity-sink blocks and two rolling blocks.Source: §5 · p. 6

5.1 Settings

• The experiments use OmniAvatar-1.3B as the causal student, initialized from ODE trajectories and then trained with on-policy DMD on 4,000 curated multimodal conditions.OmniAvatar accepts text, image, and audio inputs; ODE initialization runs for 20,000 steps, followed by about 1,000 DMD steps.Source: §5.1 · p. 6

5.2 Single-Round Evaluation

• 24.82 FPS versus 0.97 FPS and 0.33s versus 83.44s first-frame latency: the distilled model matches or exceeds OmniAvatar-1.3B on quality across three benchmarks.The comparison covers visual quality, aesthetics, and lip-sync on HDTF, AVSpeech, and CelebV-HQ.Source: §5.3 · p. 7, Table 1 · p. 7

5.3 Multi-Round Evaluation

• LiveTalk outperforms Veo3 and Sora2 on multi-video coherence and content quality while remaining competitive on other visual interaction dimensions.The evaluation uses a curated multi-round interaction benchmark.Source: §5.3 · pp. 7–8, Table 2 · p. 8

5.4 Ablations

• Curated conditions improve visual quality and synchronization, and 20,000-step ODE initialization resolves remaining blurriness and flickering and brings quality metrics to their peak.The ablation compares sequentially added components on 100 five-second HDTF clips.Source: §5.4 · p. 9, Table 3 · p. 9
• Doubling both learning rates and raising teacher-score CFG to 6 improves lip-sync at a modest visual-quality cost; further increases cause instability or oversaturation.Mouth movements become more pronounced and articulated under this setting.Source: §5.4 · p. 9

6. Conclusion

• With lower-quality conditions, visual quality remains degraded, with poor temporal consistency and abrupt color shifts between frames.This configuration applies the other improvements while reverting to baseline multimodal conditions.Source: §6 · p. 10
• Figure 4 presents generated-video examples from the different configurations as evidence of the proposed improvements.The conclusion states that these results validate improvements in generation quality and stabilization of multimodal distillation.Source: §6 · p. 10

6 Conclusion

• The recipe distills a bidirectional many-step model into a causal four-step autoregressive model, and LiveTalk streams video conditioned on text, image, and audio.The conclusion reports coherent multi-turn interactions with significantly lower latency than state-of-the-art video models.Source: §6 · p. 10

A Multi-Round Evaluation Details

• A VLM assesses nine interaction-benchmark dimensions using structured prompts, grouped into visual interaction performance and interaction content quality.The appendix also provides implementation details for the evaluation protocol.Source: A Multi-Round Evaluation Details · p. 14

A.1 Visual Interaction Performance

• Visual performance criteria cover emotional appropriateness and nonverbal interaction, including adaptive expressions, natural gaze, gestures, listening cues, and bidirectional authenticity.Emotional appropriateness includes intensity calibration and coherent emotional transitions; nonverbal interaction includes cues such as nods, pauses, and micro-expressions.Source: §A.1 · p. 14
• Multi-video coherence evaluates consistency across responses, while conversational naturalness distinguishes spontaneous human-like interaction from scripted or broadcast-style delivery.Coherence criteria include stable identity and scene, emotional continuity, and consistent behavior; naturalness includes pauses and speech hesitations.Source: §A.1 · p. 14

A.2 Interaction Content Quality

• Interaction content quality assesses whether verbal responses are contextually appropriate, directly address user queries, and provide complete, self-contained answers.Source: §A.2 · p. 14

A.2 Interaction Content Quality

• Logical Consistency checks for contradictions, factual errors, and logical inconsistencies within responses and across conversation history.Context Understanding assesses state tracking, anaphora resolution, and building on earlier turns.Source: §A.3 · p. 15
• Overall Interaction Experience provides a holistic assessment of conversational fluency, task completion, and user satisfaction.Source: §A.3 · p. 15
• Context Understanding measures whether the assistant maintains conversational state and resolves anaphoric references.It also assesses whether responses build on information established in prior turns.Source: §A.3 · p. 15

A.3 Evaluation Protocol Details

• Qwen2.5-VL-30B-A3B-Instruct scores rubric-defined dimensions using up to 32 frames per video at 784×784 resolution, with temperature 0.1 and top-p 0.9.Responses are structured for automated score parsing.Source: §A.3 · p. 15
• FunASR transcribes both 16kHz user audio and assistant video audio tracks into English text for the conversational transcripts.The protocol uses the speechparaformer-asr-en-16k-vocab4199 model.Source: §A.3 · p. 15
• For multi-video conversations, the evaluator receives responses in chronological order with the full transcript, enabling cross-video coherence and emotional-transition analysis across dialogue turns.Content-quality dimensions use the full conversation history to assess semantic coherence, logical consistency, and contextual understanding.Source: §A.3 · p. 15

B Training Configuration Details

• The distillation pipeline has two sequential stages: ODE initialization followed by on-policy distribution matching distillation (DMD).Table 4 provides the hyperparameters and optimization settings for both stages.Source: B Training Configuration Details · p. 16
• During DMD, the critic score network sψ is updated for 20 steps before training the student generator network gϕ.The critic updates are used to obtain accurate score estimates.Source: Table 4 · p. 16
• Exponential moving average updates are enabled from step 200 onward.Source: Table 4 · p. 16