Source-linked AI summary
Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation
Zhe Kong, Feng Gao, Yong Zhang, Zhuoliang Kang, Xiaoming Wei, Xunliang Cai, Guanying Chen, Wenhan Luo
TL;DR
Existing audio-driven animation largely focuses on single humans and struggles with multi-stream audio binding and instruction following in multi-person generation. MultiTalk introduces the multi-person conversational video generation task and uses L-RoPE, adaptive localization, and targeted training strategies to address these issues. It performs strongly across talking-head, talking-body, and multi-person evaluations while retaining instruction-following capability, though facial expression quality is better with real than synthesized audio.
Problem
Existing methods primarily focus on single-human animation and struggle with multi-stream audio, incorrect audio-person binding, and instruction following.
Method
MultiTalk combines L-RoPE for audio-person binding with adaptive person localization and partial-parameter and multi-task training strategies.
Results
MultiTalk demonstrates superior performance across talking-head, talking-body, and multi-person datasets, including lip synchronization, video quality, instruction following, and interactive multi-person handling.
Takeaways & Limitations
MultiTalk provides a framework for audio-driven multi-person conversational video generation with accurate audio-person binding and preserved instruction-following capability.
Takeaways & Limitations
Facial-expression performance is better with real audio than synthesized audio, potentially because training uses real audio data.
Abstract
from arXiv · showhide
Audio-driven human animation methods, such as talking head and talking body generation, have made remarkable progress in generating synchronized facial movements and appealing visual quality videos. However, existing methods primarily focus on single human animation and struggle with multi-stream audio inputs, facing incorrect binding problems between audio and persons. Additionally, they exhibit limitations in instruction-following capabilities. To solve this problem, in this paper, we propose a novel task: Multi-Person Conversational Video Generation, and introduce a new framework, MultiTalk, to address the challenges during multi-person generation. Specifically, for audio injection, we investigate several schemes and propose the Label Rotary Position Embedding (L-RoPE) method to resolve the audio and person binding problem. Furthermore, during training, we observe that partial parameter training and multi-task training are crucial for preserving the instruction-following ability of the base model. MultiTalk achieves superior performance compared to other methods on several datasets, including talking head, talking body, and multi-person datasets, demonstrating the powerful generation capabilities of our approach.
1 Introduction
Audio-driven human animation has advanced from synchronized single-person talking heads and bodies, but multi-person conversational generation introduces multi-stream audio, binding, and localization challenges. MultiTalk addresses these challenges with L-RoPE and training strategies designed to preserve instruction following.
- Motivation: Existing audio-driven animation methods mainly target single-person talking heads or bodies, rather than multi-person conversations.Talking-head methods emphasize facial expressions, while talking-body methods synchronize broader body movements.
- Challenges: Multi-person conversation requires handling multiple audio streams, binding each stream to the correct person, and adaptively localizing dynamic people.These challenges distinguish the proposed task from single-human animation.
- Approach: MultiTalk introduces L-RoPE, which assigns identical labels to audio embeddings and video latents to activate person-specific regions in audio cross-attention.The method is designed to resolve incorrect audio-person binding during multi-person generation.
- Training: The framework explores multi-stage, partial-parameter, and multi-task training, with partial-parameter and multi-task training observed as crucial for preserving instruction following.The multi-event image-to-video dataset is also described as crucial, especially with limited compute and data.
- Evaluation: Evaluations on talking-face, talking-body, and multi-person conversation datasets demonstrate the effectiveness of the proposed method.The contribution statement reports results across these dataset categories.
2 Related Work
Earlier audio-driven animation pipelines separated audio-to-motion prediction from motion-to-video rendering, while newer end-to-end diffusion methods directly couple audio and facial dynamics. Video diffusion models provide a broader generative foundation for video synthesis, but the cited approaches retain limitations in motion scope or intermediate modeling.
- Audio-Driven Human Animation: Pioneering methods use audio-to-motion models to produce intermediate representations such as 3DMM and FLAME before rendering portrait animations.GAN-based motion-to-video techniques project these representations into dynamic portraits.
- Audio-Driven Human Animation: Limitations in capturing intricate facial expressions and head movements constrain the authenticity and naturalness of early audio-to-motion pipelines.The passage identifies this as a limitation of the audio-to-motion component.
- End-to-End Synthesis: End-to-end audio-to-video methods omit intermediate representations and use a single diffusion model to integrate audio cues with facial dynamics.These methods demonstrate enhanced naturalness and consistent portrait animation capability.
- End-to-End Synthesis: End-to-end audio-to-video methods are constrained to support only head movement, motivating work on audio-driven body animation.The passage contrasts their scope with subsequent efforts targeting body animation.
- Video Diffusion Models: Video diffusion models are broadly categorized into text-to-video and image-to-video models, with early systems commonly leveraging U-Net architectures.These models extend text-to-image diffusion capabilities toward continuous video generation.
3 Method
MultiTalk extends a DiT-based image-to-video model with audio conditioning for multi-person conversational video generation. It combines audio cross-attention, adaptive localization, L-RoPE-based audio-person binding, and training strategies designed to preserve instruction following.
- Audio-conditioned architecture: MultiTalk adds layer normalization and audio cross-attention after text cross-attention in each DiT block to incorporate audio conditions.The foundational image-to-video model animates a reference image but does not natively accept audio.
- Audio-conditioned architecture: Wav2Vec extracts acoustic embeddings, and neighboring audio frames are concatenated because motion depends on preceding and succeeding audio.The compressed audio condition is aligned with the shorter latent-video sequence through an audio adapter.
- Multi-stream audio injection: Direct concatenation or separate-and-add injection schemes fail to bind each audio stream to its corresponding person, motivating adaptive L-RoPE injection.The framework investigates four multi-stream injection schemes before introducing L-RoPE.
- Multi-stream audio injection: Adaptive person localization uses reference-image-to-video self-attention similarities to identify each person’s region in generated video latents.The reference image is partitioned into person and background masks, and similarity scores assign latent tokens to these subjects.
- Multi-stream audio injection: L-RoPE assigns labels to video and audio embeddings, using label-dependent rotations so nearby labels activate corresponding audio cross-attention regions.Different label ranges distinguish persons, while the background receives a non-audio-associated label; example audio labels are 2 and 22.
- Training strategy: Two-stage training first develops single-person audio and lip synchronization, then uses dual-stream audio data for multi-human animation.With limited compute and data, partial parameter training avoids the instruction-following degradation and hand or object distortion associated with full training; adding I2V data preserves instruction following.
4 Experiments
Experiments evaluate MultiTalk on talking-head, talking-body, and multi-person settings using standard quality and synchronization metrics plus qualitative comparisons. The method generally outperforms competing approaches, preserves instruction following, and handles multi-person interactions without segment inconsistencies.
- Settings: Evaluation covers talking-head, talking-body, and multi-person datasets using FID, FVD, E-FID, Sync-C, and Sync-D.Talking-head and talking-body comparisons use conventional evaluation, while multi-person evaluation measures synchronization and expression after splitting each video into person-specific segments.
- Comparisons with Competing Methods: MultiTalk surpasses most competing methods across a majority of metrics, with superior lip synchronization and video quality.Quantitative comparisons are reported for talking-head and talking-body generation in Tables 1 and 2.
- Comparisons with Competing Methods: Only MultiTalk follows text instructions successfully in the qualitative comparison and produces fewer video artifacts.The comparison is shown in Figure 6.
- Comparisons with Competing Methods: Compared with video concatenation, MultiTalk handles interactive scenarios without inconsistencies between left and right video segments.The method also adaptively identifies each person’s localization, benefiting audio binding.
- Analyses: Multi-stream audio training achieves performance comparable to single-human animation models on talking-head and talking-body datasets.The comparison tests whether multi-stream training causes performance degradation.
- Analyses: Different label choices for persons yield comparable metrics, indicating that L-RoPE is insensitive to label-range variations.The label-range ablation is conducted on the MTHM conversation dataset and reported in Table 3.
5 Conclusion
The paper introduces the audio-driven multi-person conversational video generation task and MultiTalk framework. It reports effective audio-person binding and identifies partial parameter training and multi-task training as essential for maintaining instruction following, while synthesized audio yields weaker facial expression performance than real audio.
- Conclusion: MultiTalk addresses audio-driven multi-person conversational video generation with effective multi-stream audio injection and audio-person binding.The conclusion names the proposed L-RoPE method for this purpose.
- Conclusion: Partial parameter training and multi-task training are essential for maintaining the base model’s instruction-following ability.The conclusion associates these strategies with strong instruction-following capability.
- Conclusion: Facial expression performance is better with real audio than synthesized audio.The authors attribute the gap possibly to training with real audio and identify mitigation as future work.
A.1 Dataset Details
The appendix describes evaluation on a dual-human talking-body dataset and provides examples from the MTHM dataset.
- A.1 Dataset Details: Dual-human evaluation splits each video into left and right segments, each containing one person and the corresponding audio.Sync-C, Sync-D, and E-FID are computed on these person-specific segments.
- A.1 Dataset Details: Figure 8 visualizes examples from the collected MTHM dataset.
A.2 Sample Details
Sampling uses 40 steps and a negative prompt designed to filter undesired diffusion-model variations.
- A.2 Sample Details: All experiments use 40 sampling steps with a negative prompt targeting artifacts and low-quality or malformed outputs.The listed exclusions include blur, static content, poor quality, compression residue, and deformed anatomy.
B.1 Full Parameter Training vs Cross-attention Training
The comparison examines full parameter training against fine-tuning only the audio cross-attention layer. Full parameter training degrades instruction-following and causes hand distortion, whereas cross-attention training preserves the base model’s instruction-following ability.
- B.1 Full Parameter Training vs Cross-attention Training: Full parameter training degrades instruction-following, especially for motion and interaction, and causes hand and object distortion.The comparison is shown in Fig. 9.
- B.1 Full Parameter Training vs Cross-attention Training: Fine-tuning only the audio cross-attention layer avoids these issues and preserves the base model’s instruction-following ability.
- B.1 Full Parameter Training vs Cross-attention Training: Figure 9 compares full parameter training with cross-attention training.
B.2 Long Video Generation
MultiTalk supports long-video generation through an autoregressive-based method. The reported example contains 305 frames, with results presented in Fig. 10.
- B.2 Long Video Generation: 305 frames are included in the reported long-video generation example.The experimental results are shown in Fig. 10.
- B.2 Long Video Generation: Figure 10 presents the generation result of long videos.
C Societal Impacts
The framework is presented as an effective tool for audio-driven multi-person conversational video generation, but it could be misused to create deceptive celebrity videos.
- C Societal Impacts: Malicious entities could exploit the framework to generate fake celebrity videos that potentially mislead the public.The paper notes this concern is shared across human animation methodologies.