PaperlayerView exact revision on arXiv

Source-linked AI summary

Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversation

Taekyung Ki, Sangwon Jang, Jaehyeong Jo, Jaehong Yoon, Sung Ju Hwang

arXiv:2601.00664v2cs.LGcs.AIcs.CVcs.HCcs.MM

TL;DR

Existing talking-head systems often miss the bidirectional, expressive nature of face-to-face interaction and must also react under real-time causal constraints. Avatar Forcing combines causal diffusion forcing with preference optimization from synthetic losing samples to process multimodal user cues. It achieves roughly 500ms latency and is preferred over 80% against the strongest baseline, while its scope remains focused on head-motion interactions rather than richer bodily cues.

  • Problem

    Existing avatar models primarily generate one-way audio-synchronized motion, while real interactive avatars must respond to verbal and non-verbal cues under low-latency causal constraints.

  • Method

    Avatar Forcing uses causal diffusion forcing with key-value caching and preference optimization based on under-expressive samples created by dropping user signals.

  • Results

    500ms latency and over 80% human preference against the strongest baseline demonstrate real-time interaction and improved naturalness, responsiveness, and overall interaction quality.

  • Takeaways & Limitations

    Avatar Forcing generates reactive and expressive head motion from both verbal and non-verbal user signals for real-time human-AI communication.

  • Takeaways & Limitations

    The system focuses on head-motion latent spaces, limiting its modeling of richer bodily cues such as hand gestures and leaving some scenarios in need of more explicit gaze or emotion control.

Abstract

from arXiv · show

Talking head generation creates lifelike avatars from static portraits for virtual communication and content creation. However, current models do not yet convey the feeling of truly interactive communication, often generating one-way responses that lack emotional engagement. We identify two key challenges toward truly interactive avatars: generating motion in real-time under causal constraints and learning expressive, vibrant reactions without additional labeled data. To address these challenges, we propose Avatar Forcing, a new framework for interactive head avatar generation that models real-time user-avatar interactions through diffusion forcing. This design allows the avatar to process real-time multimodal inputs, including the user's audio and motion, with low latency for instant reactions to both verbal and non-verbal cues such as speech, nods, and laughter. Furthermore, we introduce a direct preference optimization method that leverages synthetic losing samples constructed by dropping user conditions, enabling label-free learning of expressive interaction. Experimental results demonstrate that our framework enables real-time interaction with low latency (approximately 500ms), achieving 6.8X speedup compared to the baseline, and produces reactive and expressive avatar motion, which is preferred over 80% against the baseline.

1. Introduction

Existing talking-head models mainly produce one-way, audio-synchronized motion rather than the bidirectional verbal and non-verbal behaviors of face-to-face interaction. Avatar Forcing addresses real-time causal generation and expressive interaction through diffusion forcing and preference optimization.

  • Motivation: Existing avatar models prioritize audio-synchronized lip and head motion over bidirectional conversational engagement.This focus overlooks continuous verbal and non-verbal exchanges, including nodding and empathic responses.
  • Challenges: Real-time interaction requires causal generation because methods using over 3 seconds of future conversation context introduce notable delays.The system must continuously process speech, head motion, and facial expression with low inference time and minimal latency.
  • Challenges: Expressive reaction is difficult to learn because interaction behavior is ambiguous, weakly annotated, and often represented by stiff, low-variant training motions.The difficulty is especially pronounced for responses to non-verbal cues.
  • Approach: Avatar Forcing uses causal diffusion forcing in a learned motion latent space and reuses past information through key-value caching for low-latency multimodal generation.The framework continuously processes user inputs instead of waiting for future conversation context.
  • Approach: Synthetic under-expressive motion created by dropping user signals serves as less-preferred data for preference optimization without additional human annotations.This training strategy targets more expressive and reactive avatar motion.

2. Related Works

Prior work generated lip-synchronized, holistic, or listening motions, but interactive behavior remains difficult because responses are context-dependent and one-to-many. Avatar Forcing models both participants’ verbal and non-verbal influence continuously with diffusion forcing.

  • Prior avatar generation: Earlier talking-avatar methods progressed from audio-driven lip motion to holistic head movements and facial expressions.Examples include rhythmical head motion, eye blinking, jaw movement, and 3D morphable-model representations.
  • Listening motion: Listening-motion methods generate nodding or focusing, but the speaker-listener relationship is one-to-many and context-dependent under weak supervision.Some approaches use personalized motion, text instruction, or pose priors to address this challenge.
  • Dyadic motion: Dyadic motion methods model conversational participants, but discrete verbal and non-verbal spaces can require manual role switching and cause discontinuous transitions.This limitation motivates continuous modeling of interactive behavior.
  • Avatar Forcing: Avatar Forcing continuously models both participants’ verbal and non-verbal influence while generating talking, head movement, listening, and focusing motions.Its diffusion-forcing framework is designed to respond promptly to multimodal user signals.

3. Background

Diffusion forcing provides a causal sequential-generation formulation by independently noising sequence tokens and regressing a vector field toward a target field. Direct preference optimization aligns generated samples using preferred and less-preferred outcomes without an explicit reward model.

  • Diffusion forcing: Diffusion forcing predicts the next token conditioned on noisy past tokens as an efficient sequential generative model.It reformulates conventional teacher forcing for causal sequence generation with diffusion guidance.
  • Diffusion forcing: Each sequence token receives an independent noise level, producing an independently noised token sequence for training.The noise levels are represented by t := (t1, t2, · · ·, tN).
  • Diffusion forcing: The diffusion-forcing objective regresses the model vector field vθ toward the target vector field vn.The objective is defined over the independently noised tokens and their noise levels.
  • Direct Preference Optimization: Direct Preference Optimization aligns a model with human preferences without explicitly training a reward model.Its objective compares preferred and less-preferred samples relative to a frozen reference model using a sigmoid-based deviation term.

4. Avatar Forcing

Avatar Forcing generates interactive avatar motion from multimodal user and avatar signals in a causal motion-latent framework, combining unified conditioning with blockwise diffusion forcing. Its look-ahead causal mask addresses cross-block jitter, while preference optimization targets more expressive reactions without additional human labels.

  • Framework overview: Avatar Forcing conditions avatar motion generation on user audio, user motion, and avatar audio.The framework models motion latents autoregressively from past motion and condition triplets containing user and avatar audio.
  • Framework overview: The Dual Motion Encoder integrates multimodal user signals with avatar audio into a unified user-avatar condition.Cross-attention first aligns user signals and then relates them causally to avatar audio.
  • Causal motion generation: The causal DFoT generator predicts motion blocks without future context, unlike the bidirectional architecture used in INFP.Blockwise attention preserves local bidirectional dependencies within blocks while enforcing causal dependencies across blocks and supporting KV caching.
  • Causal motion generation: Look-ahead causal attention allows limited future-frame access to reduce severe per-frame motion jitter while preserving overall causality.The mask uses frame indices, block size B, and look-ahead frame size l to constrain attention across blocks.
  • Expressive motion optimization: Preference optimization enhances expressive interaction by treating motion generated after dropping user signals as less-preferred samples.This approach targets the ambiguity of appropriate reactions to user motion and audio without requiring additional human annotations or a dedicated reward model.

5. Experiments

Experiments evaluate Avatar Forcing across interactive, talking, listening, and ablation settings, showing lower latency, stronger reactive and expressive motion, and competitive visual and lip-synchronization quality. Human studies and ablations further support the roles of multimodal user motion and preference optimization.

  • Interactive Avatar Evaluation: Avatar Forcing achieves 0.5s latency on RealTalk while maintaining Visual Quality and Lip Synchronization comparable to INFP*; INFP* has 3.4s latency.The comparison uses reproduced INFP* results and includes a non-interactive talking-head model for reference.
  • Interactive Avatar Evaluation: Avatar Forcing significantly outperforms INFP* in Reactiveness and Motion Richness on RealTalk, producing more reactive and expressive motion.Qualitative comparisons visualize stronger reactions and expressiveness in the generated avatar motion.
  • Interactive Avatar Evaluation: Over 80% of human-evaluation judgments prefer Avatar Forcing overall, with strong preferences across the evaluated perceptual metrics.The study compares Avatar Forcing with INFP* using 42 participants and 12 video sets.
  • Comprehensive Analysis: Avatar Forcing achieves competitive talking-head performance on HDTF and the best image and video quality among the compared models.The comparison evaluates FID, FVD, CSIM, LSE-D, and LSE-C against SadTalker, Hallo3, FLOAT, and INFP*.
  • Comprehensive Analysis: On ViCo, Avatar Forcing outperforms listening-head baselines on almost all metrics and achieves the best user–avatar synchronized motion generation measured by rPCC.The evaluation uses expression and pose distance, rPCC, SID, and Var metrics.
  • Ablation Study: Removing user motion reduces Reactiveness and Motion Richness, while the full model responds to smiles and speech with corresponding avatar behavior.Without user motion, the avatar remains static during silent audio; with it, the avatar smiles after the user smiles and becomes more focused when the user speaks.
  • Ablation Study: Preference optimization improves rPCC-Exp, rPCC-Pose, SID, and Var, yielding more diverse facial expressions, head movement, and responses to the user’s smile.The comparison is against a variant without fine-tuning, whose motion is less diverse and interactive.

6. Conclusion

The paper concludes that Avatar Forcing is a real-time interactive head-avatar model based on diffusion forcing. It generates reactive and expressive motion from verbal and non-verbal user signals and is presented as a step toward interactive virtual avatars.

  • Conclusion: Avatar Forcing is a real-time interactive head-avatar model based on diffusion forcing that uses verbal and non-verbal user signals.The conclusion identifies the model’s interaction modality and generative foundation.
  • Conclusion: The paper presents Avatar Forcing as a step toward truly interactive virtual avatars and real-time human-AI communication.Further discussion of ethics, limitations, and future work is deferred to Appendix E.
  • Background: Flow matching transforms a prior distribution into a target data distribution by solving an ordinary differential equation.The vector field vθ defines the ODE trajectory from the prior to the target distribution.

B. Details on Model Architecture

The model combines a motion-latent representation with dual multimodal encoding and a causal diffusion-forcing generator. Its attention masks and offset strategy support temporally consistent, low-latency blockwise inference, while preference optimization targets more expressive motion.

  • Motion latent representation: The motion latent auto-encoder decomposes encoded images into identity and motion representations for reconstructing driving images.It is trained using source and driving images sharing the same identity.
  • Multimodal conditioning: The Dual Motion Encoder integrates user motion, user audio, and avatar audio through two cross-attention layers.User motion is queried first, then the aligned motion is integrated with avatar audio; each layer uses four heads with hidden dimension d = 512.
  • Causal motion generation: The motion generator uses eight DFoT transformer blocks followed by a transformer head, with noisy latents modulated by per-frame flow time.The architecture applies shared AdaLN scale-shift coefficients through a time-embedding layer.
  • Causal motion generation: Blockwise causal look-ahead attention supports causal motion generation and improves temporal consistency across generated videos.The look-ahead design is paired with sliding-window attention for aligning driving signals to noisy latents.
  • Preference optimization: Preference optimization compares preferred and less-preferred motion latents using diffusion-forcing vector-field objectives relative to a reference model.The formulation defines noisy preferred and less-preferred latents at per-frame flow times before applying the DPO loss.
  • Real-time inference: The inference procedure divides frames into blocks, solves the ODE for each block, and decodes the resulting motion latents into video frames.KV caching and offsets reuse prior clean motion latents and conditions while supplying look-ahead context.
  • Real-time inference: The offset strategy supplies the last l clean motion latents from the previous block to avoid the additional latency caused by look-ahead attention.This preserves look-ahead context while generating the current block with separate flow-time schedules for offset and noisy frames.

D.2. Training Details

Training freezes the motion latent auto-encoder while optimizing the vector-field model, then applies a short preference-optimization fine-tuning stage. The evaluation baseline is reproduced on the same motion-latent space because official baseline code is unavailable.

  • Training: The vector-field model is trained for 2000k steps with the motion latent auto-encoder frozen and an L1 distance objective.Preference fine-tuning uses λ = 0.1 and β = 1000, with the reference model initialized from the trained model.
  • Preference optimization: Preference optimization fine-tunes the vector-field model for 5k steps, after which additional tuning yields no further performance gains.The reference vector field uses the same weights as the trained model.
  • Baseline: Because official baseline implementations are unavailable, the authors reproduce INFP on the motion latent space and call it INFP*.The reproduction follows the original core module and uses a bidirectional Transformer with a 75-frame window plus 10 context frames.

D.4. Additional Ablation Studies

Ablation studies show that diffusion forcing is more robust than autoregressive diffusion over long horizons, while motion block size trades latency against temporal consistency and lip synchronization. Additional videos expose motion jitter that quantitative metrics may miss.

  • Autoregressive comparison: Diffusion forcing is more robust to long-horizon motion drift than autoregressive diffusion.The comparison trains autoregressive diffusion with four clean context blocks and one noisy block.
  • Motion block size: Larger motion blocks reduce latency, whereas smaller blocks improve FVD and LSE-D at higher latency.The ablation fixes the number of training frames at N = 50.
  • Additional metrics: The additional ablations evaluate visual quality and lip synchronization, alongside user-motion and preference-optimization variants.The reported metrics include Visual Quality and Lip Synchronization, with accompanying video results.
  • Attention-mask ablation: Using only a blockwise causal mask produces visible motion jitter that is difficult to capture with quantitative metrics.The attention-mask ablation compares masking methods through additional video results.

D.5. Evaluation Metrics

The evaluation measures avatar motion reactiveness, richness, visual quality, identity preservation, and lip synchronization using 3DMM parameters, image or video features, and audio-visual embeddings. Human evaluation compares generated videos across five metrics.

  • Motion metrics: Reactiveness and motion richness use SPECTRE-extracted 3DMM expression and pose parameters from user, ground-truth avatar, and generated avatar videos.The 3DMM representation contains 50 expression dimensions and 6 pose dimensions.
  • Motion metrics: rPCC measures the discrepancy between generated and ground-truth motion synchronization correlations with user parameters.The metric compares PCC(y|x) and PCC(ŷ|x) using an L1 difference.
  • Visual quality: FID, FVD, and CSIM assess image quality, video quality, and identity preservation, respectively.FID and FVD compare generated visual features, while CSIM compares facial embeddings with the source image.
  • Lip synchronization: LSE-D and LSE-C evaluate audio-lip alignment, with lower LSE-D and higher LSE-C indicating better synchronization.Both metrics derive from a SyncNet-based audio-visual synchronization model.
  • Human evaluation: Human evaluation asked 42 participants to compare 12 videos across five evaluation metrics.Participants received a reference test and answer sheet to improve evaluation consistency.

D.7. Supplementary Visual Results

The supplementary results compare Avatar Forcing with prior methods for interactive, talking-head, and listening-head avatar generation. Across these comparisons, it remains competitive while reflecting user signals and producing more expressive facial expressions.

  • Interactive Head Avatar: Avatar Forcing is compared with INFP [70] using demonstration videos for interactive head-avatar generation.The supplementary materials provide video comparisons against official INFP demonstrations.
  • Talking Head Avatar: Avatar Forcing produces competitive talking-head results while successfully reflecting user signals.The comparison includes SadTalker, Hallo3, FLOAT, and INFP* [70].
  • Listening Head Avatar: Avatar Forcing produces competitive listening-head results with more expressive facial expressions.The comparison includes RLHG, L2L, DIM, and INFP* [70].

E. Discussion

The discussion highlights both the applications and risks of interactive head-avatar generation. It also identifies scope boundaries in bodily cues, controllability, and exposure-bias handling, while suggesting directions for future extensions.

  • Ethical Consideration: Interactive head-avatar videos may support virtual avatar chat, virtual education, and other communication tools, but they also pose identity-spoofing and malicious-deepfake risks.The authors suggest watermarks, restricted licensing, and use of generated data for deepfake detection as mitigations.
  • Limitation and Future Work: The head-motion latent space limits modeling of richer bodily cues such as hand gestures.The authors identify incorporating additional user signals as a future extension.
  • Limitation and Future Work: Some scenarios require more explicit control over eye gaze or emotional shifts, and exposure bias remains incompletely addressed.The proposed future directions include eye-tracking or emotion-tracking inputs and further work in the motion latent space.
Loading 2601.00664v2…