Source-linked AI summary
AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement
Zhizhou Zhong, Yicheng Ji, Zhe Kong, Yiying Liu, Jiarui Wang, Jiasun Feng, Lupeng Liu, Xiangyi Wang, Yanjia Li, Yuqing She, Ying Qin, Huan Li, Shuiyang Mao, Wei Liu, Wenhan Luo
TL;DR
Multi-person talking-video generation faces costly data collection and difficulty modeling coherent interactions across identities. AnyTalker uses iterative identity-aware attention and a two-stage training strategy based largely on single-person videos, then refines interactions with limited multi-person data. The authors report a favorable balance of scalability, interactivity, and lip synchronization, alongside a benchmark and metric for interaction evaluation.
Problem
Multi-person video generation requires costly curated data, and existing approaches often struggle to produce natural interactions across identities.
Source: §1 · p. 2Method
AnyTalker iteratively processes identity-audio pairs with shared attention and combines single-person training data with limited authentic multi-person refinement.
Source: §1 · p. 2, §3.2 · p. 4Results
Takeaways & Limitations
The paper contributes a two-person interaction benchmark and an eye-keypoint-activity metric for evaluating interactivity during listening.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Recently, multi-person video generation has started to gain prominence. While a few preliminary works have explored audio-driven multi-person talking video generation, they often face challenges due to the high costs of diverse multi-person data collection and the difficulty of driving multiple identities with coherent interactivity. To address these challenges, we propose AnyTalker, a multi-person generation framework that features an extensible multi-stream processing architecture. Specifically, we extend Diffusion Transformer's attention block with a novel identity-aware attention mechanism that iteratively processes identity-audio pairs, allowing arbitrary scaling of drivable identities. Besides, training multi-person generative models demands massive multi-person data. Our proposed training pipeline depends solely on single-person videos to learn multi-person speaking patterns and refines interactivity with only a few real multi-person clips. Furthermore, we contribute a targeted metric and dataset designed to evaluate the naturalness and interactivity of the generated multi-person videos. Extensive experiments demonstrate that AnyTalker achieves remarkable lip synchronization, visual quality, and natural interactivity, striking a favorable balance between data costs and identity scalability.
1. Introduction
Multi-person talking-video generation must handle interactions and distinct identities, but prior approaches face high data-collection costs and often produce unnatural interactions. AnyTalker combines scalable identity driving with training from mostly single-person videos, limited real multi-person refinement, and new tools for evaluating interactivity.
1. Introduction
- Prior multi-person methods require hundreds to thousands of hours of curated data, while existing datasets often focus on single-person monologues or isolated facial animation.Turn-taking, role-switching, and nonverbal cues such as eye gaze complicate data annotation; existing approaches also often struggle to model genuine interactivity.Source: §1 · p. 2
- The paper introduces a benchmark of two-person speech and eye-contact videos with speaking/listening labels, plus an eye-keypoint-activity metric for evaluating interactivity during listening.The benchmark and metric address the lack of multi-person interaction assessment in commonly used single-person benchmarks.Source: §1 · p. 2
- AnyTalker pairs an extensible multi-stream architecture with a two-stage pipeline, using single-person data to learn multi-speaker patterns and limited multi-person data to refine interactions.The framework supports arbitrary numbers of drivable identities and is designed to balance scalability, interactivity, lip synchronization, and data cost.Source: §1 · p. 2
- Experiments are reported to show state-of-the-art performance and a favorable balance among identity scalability, interactivity, lip synchronization, and data cost.Source: §1 · p. 2
2. Related Work
Prior audio-driven talking-video work has advanced single-person generation, while multi-person methods add identity and audio control but remain constrained by data costs and limited interaction. AnyTalker explores learning multi-person speaking patterns from single-person data with an extensible multi-stream architecture.
2.1. Audio-driven Talking Video Generation
- Audio-driven video methods build on diffusion and video foundation models for improvements such as long-video generation and synchronized lips, but most still target one person or produce restricted multi-person interactions.The cited methods also report full-body synthesis, non-human generation, and coherent hand movements.Source: §2.1 · pp. 2–3
2.2. Multi-person Video Generation
- Multi-person systems use mechanisms such as identity embedding routers and audio-person binding, but data-intensive approaches require hundreds to thousands of hours and single-person-trained systems may yield fragmented interactions.AnyTalker investigates learning multi-person speaking patterns from single-person data with extensible multi-stream audio processing and low training-data cost.Source: §2.2 · p. 3
3. Method
AnyTalker extends a video diffusion model with Audio-Face Cross Attention (AFCA) for processing multiple identity-audio streams. Its training combines single-person data, including synthetic horizontal concatenations, with a small amount of authentic multi-person data for interaction refinement.
3. Method
- The framework adds AFCA to the Wan I2V-based model to process multiple audio and identity inputs, and divides training into two data-dependent stages.Source: §3 · pp. 3–4
3.1. Preliminaries
- The model encodes video, text, reference-image, and audio features using video patchification, T5, CLIP, and Wav2Vec2, respectively.Attention layers connect to the final output FFN layer, as in Wan.Source: §3.1 · p. 4
3.2. Audio-Face Cross Attention
- AFCA iterates over each face-audio pair with shared parameters and sums the outputs, allowing the number of drivable identities to scale arbitrarily.A temporal mask aligns each subsequent video latent frame with a local window of four audio tokens, while the first latent frame attends to all audio tokens. A precomputed global face-bounding-box mask constrains the output to the face region.Source: §3.2 · p. 4
3.3. Training Strategy
- Training uses single-person clips and horizontal concatenations with a 50% probability for each mode, then refines interaction using authentic multi-person data.The single-person stage supports lip synchronization and generalized speaking patterns; omitting single-person data degrades accurate lip movement. The real-data refinement uses 12 hours of quality-controlled dual-person clips.Source: §3.3 · p. 5
4. Interactivity Evaluation
The section addresses the lack of interaction-focused multi-person benchmarks by presenting a two-person dataset and an eye-motion metric that evaluates responsiveness during listening periods.
4. Interactivity Evaluation
- Common single-person talking-head benchmarks lack multi-person interactions, while InterActHuman’s test set covers only single-speaker scenarios.The section identifies these limits as reasons existing benchmarks are unsuitable for assessing natural character interactions.Source: §4 · p. 5
4.1. Dataset Construction
- InteractiveEyes contains roughly 10-second videos of two people, manually segmented to capture speaking, listening, mutual gaze, and head movements.The videos include varied eye interactions and authentic references.Source: §4.1 · p. 6, §4 · p. 6
4.2. Proposed Interactivity Metric
- The metric tracks motion amplitude in face-aligned eye keypoints, using their sequence across generated frames to assess natural interaction.Motion is evaluated during listening periods because listening subjects can appear rigid even when speaking subjects are well driven.Source: §4.2 · p. 6
- The eye-focused Interactivity metric aligns with human perception: sluggish eye movement scores lower, while head turns and eyebrow raises score higher.An exclusion algorithm is used to avoid misjudging abnormal eye movements.Source: §4.2 · p. 6
5. Experiments
Experiments compare AnyTalker with competing methods on single- and multi-person benchmarks, then ablate its components and training data. The multi-person comparisons and ablations examine lip synchronization, generation quality, identity similarity, and interactivity.
5. Experiments
- Training uses roughly 1,000 hours of single-person video for stage one and about 12 hours of filtered two-person clips for stage two.Evaluation includes HDTF, VFHQ, and a self-collected multi-person conversation dataset, with 20 videos selected from each benchmark and identities held out from training.Source: §5 · p. 6
- Multi-person assessment uses Interactivity as its primary metric and adjusts Sync-C to score lip synchronization during speaking phases.The adjustment avoids long listening segments influencing the final lip-synchronization score.Source: §5 · pp. 6–7
5.1. Comparison with SOTA methods
- On HDTF and VFHQ, AnyTalker achieves the best or competitive results across all single-person metrics.The 1.3B model outperforms AniPortrait, EchoMimic, and StableAvatar in lip synchronization despite their similar parameter counts.Source: Table 1 · p. 7, §5.1 · p. 7
- Both AnyTalker models achieve the best Interactivity results on InteractiveEyes, and the 14B model achieves the best results across all metrics.The comparison includes Bind-Your-Avatar and MultiTalk.Source: Table 2 · p. 7, §5.1 · p. 7
- Under matched prompts, reference images, and audio streams, AnyTalker produces more natural eye and head interactions than Bind-Your-Avatar and MultiTalk.MultiTalk has weaker eye interaction, while Bind-Your-Avatar tends toward more static expressions; the comparison includes real and generated reference images.Source: §5.1 · p. 7, Figure 6 · p. 8
5.2. Ablation Study
- Ablations find that Audio-Face Cross Attention, the attention-output mask token, and concatenated multi-person data each contribute to model performance.Concatenated data is most beneficial to lip-movement accuracy; the full model's slightly higher FID is attributed to more dynamic actions.Source: §5.2 · p. 8, Table 3 · p. 8
- On InteractiveEyes, models trained with concatenated single-person data outperform those without it across evaluation dimensions, especially lip synchronization and interactivity.The ablation also reports unstable outputs when authentic single-person data is not mixed in stage one.Source: Table 4 · p. 8, §5.2 · pp. 8–9
- After authentic multi-person fine-tuning, models with concatenated data in stage one perform better, with a slight lip-synchronization decrease but significantly improved interactivity.The authors characterize this as a reasonable trade-off.Source: §5.2 · p. 9
6. Conclusion
AnyTalker combines an extensible multi-stream architecture with training that uses concatenated single-person data to learn multi-person speaking patterns. Experiments report a balance of lip synchronization, identity scalability, and interactivity in multi-person scenarios.
6. Conclusion
- AnyTalker introduces Audio-Face Cross Attention, an extensible structure for identity scaling and cross-identity interactions.The framework uses concatenation-based augmentation to leverage single-person data for learning multi-person speaking patterns.Source: §6 · p. 9
- The paper proposes an interactivity evaluation metric and a tailored benchmark for assessing multi-person videos.Source: §6 · p. 9
- Experiments suggest AnyTalker balances lip synchronization, identity scalability, and interactivity in multi-person scenarios.Source: §6 · p. 9
Supplementary Material
The supplementary material includes a SyncNet Score Matrix for two-person data.
Supplementary Material
- Figure 7 presents a SyncNet Score Matrix for two-person data.Source: Figure 7 · p. 13
Outline
The supplementary material contains experimental details and extended experiments on AnyTalker.
Outline
- The supplement is organized into two sets of additional information on AnyTalker.Source: Outline · p. 13
A. Experimental Details
- Experimental details cover training data processing, inference data construction, hyperparameters, and inference settings.Source: A. Experimental Details · p. 13
B. Extended Experiments
- Extended experiments add analysis of the Interactivity metric, experimental results, and interactivity-refinement effectiveness.Source: B. Extended Experiments · p. 13
A.1. Training Data Processing
The data pipeline filters and prepares single- and two-person talking clips, then uses concatenated single-person clips to simulate multi-person scenes. It retains about 1,000 hours of single-person data and 12 hours of two-person data after filtering.
A.1. Training Data Processing
- Single-person clips are filtered for stable, synchronized speech, segmented into utterances, and concatenated into training sequences averaging 4 s.Retained utterances average about 2 s before concatenation; resulting sequence lengths follow a Gaussian distribution with a 4 s mean and 0.5 s standard deviation.Source: §A.1 · pp. 13–14
- The pipeline retains approximately 1,000 h of high-quality single-person data and 12 h of clean two-person data after filtering for faces, speech, and audio-visual synchrony.Two-person clips also require stable left–right identity ordering and voice–face correspondence.Source: §A.1 · p. 14
- Two-person clips must contain exactly two faces in most frames, have stable left–right identity ordering, and pass diagonal voice–face SyncNet checks.Speaker activity is constrained to both speakers active or only one active, with a synchrony threshold matching the single-person setting.Source: §A.1 · p. 14
- For first-stage training, randomly selected single-person clips are horizontally concatenated after face-centered cropping to preserve a large facial region.For 480P footage, the initial crop is 480 × 416, with augmentation enlarging the crop while preserving that aspect ratio.Source: §A.1 · p. 14
- Each training sample stores video, cleaned audio, and extracted features in a shared dictionary-style structure, with the number of face–speech entries indicating one or two speakers.Audio and text features are extracted with Wav2Vec-2 and T5, respectively; text inputs are padded or truncated to 512 tokens.Source: §A.1 · p. 14
A.2. Benchmark Data Processing
The appendix specifies benchmark sampling and training settings, then analyzes cues and safeguards used in the Interactivity metric. The evaluation setup includes held-out identities and manually annotated speaking intervals for two-person videos.
A.2. Benchmark Data Processing
- Twenty clips are randomly selected from each single-person benchmark, with identities excluded from AnyTalker training and audio fixed to 6 s for GPU-memory constraints.The single-person benchmarks are HDTF and VFHQ; the two-person test videos contain exactly two faces and are approximately 10 s long.Source: §A.2 · p. 14
- InteractiveEyes includes both-speaker speech in 80% of cases and one-speaker speech with a listening partner in 20%, with speaking intervals manually annotated for Interactivity scoring.Two-person videos use stable cameras and exactly two faces throughout.Source: §A.2 · pp. 14–15
A.3. Implement Details
- The 14B model uses 32 NVIDIA H200 GPUs, with batch size 32 for 2.4M first-stage steps and batch size 16 for 50K second-stage steps.The 1.3B model uses 8 H200 GPUs, batch size 48, and the same total training steps; encoders and the 3D VAE remain frozen while the DiT network is trainable.Source: §A.3 · p. 15
B.1. More analysis about Interactivity Metric
- Eyebrow raises, head nods, head turns, and gaze shifts are presented as listener behaviors contributing to a high Interactivity score.Figure 11 illustrates four such cases, while the score is computed over the entire video rather than reported frame by frame.Source: §B.1 · p. 15, Figure 11 · p. 15
- The metric suppresses implausible motion by freezing facial landmarks when their mean displacement exceeds 10 pixels between consecutive frames.The faces are pre-aligned to a 256 × 256 canvas; the rule addresses exaggerated movements such as Bind-Your-Avatar's “lying-down” action.Source: §B.1 · pp. 15–16, Figure 12 · p. 16
B.2. Additional Experimental Results
On EMTD, AnyTalker-14B leads on three metrics and is only marginally behind MultiTalk on identity preservation. Additional results show natural inter-identity eye contact after multi-person refinement, while camera motion remains limited to rudimentary text-prompt control.
B.2. Additional Experimental Results
- AnyTalker-14B achieves the best scores on three EMTD metrics and is only marginally behind MultiTalk on identity preservation.The comparison covers half-body sequences that include hands.Source: §B.2 · p. 16
- AnyTalker consistently produces clear dentition and accurate lip movement on the HDTF and VFHQ qualitative comparisons.The comparisons include EchoMimic, StableAvatar, Sonic, MultiTalk, and OmniHuman-1.5.Source: §B.2 · p. 18
- Additional multi-person outputs cover real photographs, AIGC images, cartoons, and varying numbers of identities, with natural, context-appropriate interactions reported across these inputs.Figure 14 presents further generated animations.Source: Figure 14 · p. 17, §B.2 · p. 18
B.3. Effectiveness of Interactivity Refinement
- Refining with authentic multi-person data yields more natural eye contact between identities; single-person-only training can activate each face but leaves non-speaking identities blank.The blank non-speaking behavior is described as highly unnatural in conversational scenes.Source: §B.3 · p. 18, Figure 15 · p. 18
B.4. Future Work
- AnyTalker currently supports only rudimentary camera motions driven by textual prompts.The paper identifies additional conditional signals such as camera trajectories as a future direction.Source: §B.4 · p. 18