INFP: Audio-Driven Interactive Head Generation in Dyadic Conversations

• Prior audio-driven methods mainly model either speaking or listening, while existing dyadic approaches require manual roles and cannot switch smoothly.Source: §1 · pp. 1–2
• INFP dynamically alternates the driven portrait between speaking and listening states from dyadic audio, avoiding manual role assignment and explicit role switching.Source: §1 · p. 2
• The Motion-Based Head Imitation stage extracts communicative behaviors into a motion latent space and uses those codes to animate a static portrait.Source: §1 · p. 2
• The Audio-Guided Motion Generation stage maps dual-track dyadic audio to motion latent codes through denoising for interactive head generation.Source: §1 · p. 2
• DyConv provides large-scale, rich dyadic conversations collected from the Internet to support this research.Source: §1 · p. 2

2.1. Single-Sided Audio-Driven Head Generation

• Single-sided talking-head methods synchronize facial animation with speaker audio, whereas listening-head methods model listener responses but largely restrict them to non-verbal movement.Source: §2.1 · pp. 2–3

2.2. Head Generation in Dyadic Conversations

• Dyadic methods explore multi-turn data, joint speaker-listener modeling, or avatar synthesis, but retain explicit role handling, separate downstream generators, or high-level textual conditioning.Source: §2.2 · p. 3
• Some other dyadic interaction studies are person-specific and therefore have limited generalization to arbitrary individuals.Source: §2.2 · p. 3

3. Method

• INFP first compresses conversational behaviors into a low-dimensional motion space and then maps agent and partner audio to those codes through denoising.Source: §3 · p. 3, Figure 3 · p. 4

3.1. Motion-Based Head Imitation

• The imitation stage uses implicit motion encoding, low-dimensional descriptors, masked facial regions, and contour information to reduce appearance entanglement.Source: §3.1 · p. 3
• The motion-flow pipeline extracts source and driving motion codes, predicts a source-to-driving flow, warps source appearance features, and decodes the resulting portrait.Source: §3.1 · p. 4

3.2. Audio-Guided Motion Generation

• Dual-track audio removes explicit role assignment by constructing speaking-listening mixed motions from both conversational participants.Source: §3.2 · p. 4
• Learnable verbal and non-verbal memory banks select speaking or listening motion features according to which audio track is informative.Source: §3.2 · pp. 4–5
• A motion style vector carries global information such as emotions and attitudes, while training uses null-style and feature-dropout strategies plus single-sided warm-up clips.Source: §3.2 · p. 5
• A conditional diffusion transformer maps interactive features into stage 1 motion codes, using temporal conditioning for smooth window transitions and a lightweight design for real-time interaction.Source: §3.2 · p. 5

• Earlier datasets range from 0.4 to 115 hours and suffer from limited multi-turn coverage, monotonous backgrounds, or low face resolution.Source: §4 · p. 5
• DyConv contains over 200 hours of multi-turn conversations spanning diverse emotions and expressions, retaining frames with both visible faces and facial resolution above 400 × 400.Source: §4 · p. 5
• The dataset separates the two speakers’ audio and uses active speaker detection to match each audio stream to its corresponding face.Source: §4 · p. 5

5.2.1. Interactive Head Generation

• INFP outperforms DIM across interactive-generation metrics, including visual quality, audio-lip synchronization, identity preservation, and motion diversity.The reported gains are attributed to the disentangled expressive latent space, interactive motion guider, and conditional denoising transformer.Source: §5.2.1 · pp. 6–7, Table 1 · p. 8
• INFP produces more diverse head poses, vivid facial expressions, audio-aligned behaviors, and adaptive listening-speaking responses than DIM in qualitative comparisons.A user study with 20 participants also reports higher MOS for naturalness, motion diversity, audio-visual alignment, and visual quality.Source: Figure 4 · p. 6, §5.2.1 · p. 7, Table 2 · p. 8

5.2.2. Listening Head Generation

• INFP outperforms other state-of-the-art methods on most listening-head metrics, with effective expression and head-pose diversity and closer correspondence to real human behavior.The comparison uses ViCo and evaluates FD, MSE, SID, and Var for expression and head pose.Source: §5.2.2 · p. 7, Table 3 · p. 8

5.2.3. Talking Head Generation

• On HDTF talking-head generation, INFP leads in audio-lip synchronization, identity preservation, and FID, while trailing EchoMimic slightly in SSIM and PSNR.The paper relates the synchronization and identity results to its disentangled expressive motion latent space.Source: §5.2.3 · p. 7, Table 4 · p. 8

5.3. Ablation Study

• Removing motion memory banks causes inconsistent lip-related behavior and lower SID and Var, indicating that the banks store motion patterns across conversational states.The ablation removes both verbal and non-verbal memory banks from the interactive motion guider.Source: §5.3 · p. 7
• Removing style modulation reduces output authenticity and vividness, while replacing the hybrid facial representation degrades generation quality and leaks appearance information.These findings support the roles of style modulation and the hybrid facial representation in the two-stage framework.Source: §5.3 · pp. 7–8

5.4. Visualization of Style Control

• A style vector extracted from portrait videos controls the generated individual’s emotions or attitudes, with happier source clips producing corresponding emotional changes.The effect is illustrated using three video clips with varying degrees of happiness.Source: §5.4 · p. 8, Figure 5 · p. 8

6. Conclusion

• INFP provides a unified framework that adapts conversational roles without explicit assignment or switching, and experiments and ablations demonstrate effectiveness over other SOTA methods.The framework conditions head generation on dyadic audio and targets facial behaviors from human-human interactions.Source: §6 · p. 8
• The method currently uses conversation audio as its input, leaving visual and textual control signals as an avenue for added capacity.Source: §6 · p. 8
• INFP focuses on head synthesis, so extending generation to the upper body or whole body remains future work.Source: §6 · p. 8
• Potential misuse for fabricating deceptive talks and speeches leads the authors to restrict core-model access to research institutions.Source: §6 · p. 8