PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal-driven Talking Face Generation via a Unified Diffusion-based Generator

Chao Xu, Shaoting Zhu, Junwei Zhu, Tianxin Huang, Jiangning Zhang, Ying Tai, Yong Liu

arXiv:2305.02594v2cs.CV

TL;DR

Existing talking-face methods underuse text emotion and often rely on source-oriented generators with unstable GAN training. TGDM combines CLIP-informed text control with target-oriented texture transfer and diffusion, and experiments report superiority across several face-manipulation tasks.

  • Problem

    Existing pipelines often use task-specific generators and unstable GAN training, while prior work underuses text for emotion control.

    Source: §1 · p. 1

  • Method

    TGDM uses CLIP-informed text emotion and multi-conditional diffusion to transfer source texture onto a rendered target geometry, with an adaptation for face swapping.

    Source: §1 · pp. 1–2

  • Results

    Experiments are reported to demonstrate TGDM’s superiority over state-of-the-art methods across several face-manipulation tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TGDM supports flexible text-based emotion control and a shared target-oriented texture-transfer approach for multimodal talking-face generation and face swapping.

    Source: §7 · p. 11

    Under T = 1000 DDPM sampling, generation takes about 45 ms per face on one V100 GPU, and the model supports only 256 × 256 images.

    Source: §6 · p. 11

Abstract

from arXiv · show

Multimodal-driven talking face generation refers to animating a portrait with the given pose, expression, and gaze transferred from the driving image and video, or estimated from the text and audio. However, existing methods ignore the potential of text modal, and their generators mainly follow the source-oriented feature rearrange paradigm coupled with unstable GAN frameworks. In this work, we first represent the emotion in the text prompt, which could inherit rich semantics from the CLIP, allowing flexible and generalized emotion control. We further reorganize these tasks as the target-oriented texture transfer and adopt the Diffusion Models. More specifically, given a textured face as the source and the rendered face projected from the desired 3DMM coefficients as the target, our proposed Texture-Geometry-aware Diffusion Model decomposes the complex transfer problem into multi-conditional denoising process, where a Texture Attention-based module accurately models the correspondences between appearance and geometry cues contained in source and target conditions, and incorporate extra implicit information for high-fidelity talking face generation. Additionally, TGDM can be gracefully tailored for face swapping. We derive a novel paradigm free of unstable seesaw-style optimization, resulting in simple, stable, and effective training and inference schemes. Extensive experiments demonstrate the superiority of our method.

1 INTRODUCTION

Existing talking-face generators often rearrange source features, use unstable GAN training, and leave text emotion underused. TGDM instead uses text-controlled, target-oriented texture transfer with diffusion, and extends this paradigm to face swapping.

1 INTRODUCTION

  • Source-oriented generators can lose identity and background, produce warping artifacts under large appearance changes, and remain task-specific and vulnerable to GAN instability.The introduction also identifies underuse of text for emotion control as a separate gap.Source: §1 · p. 1
  • TGDM uses CLIP-informed text emotion and multi-conditional diffusion for target-oriented transfer of source texture onto rendered geometry.The text representation is intended to support flexible control, including unseen emotions.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • For face swapping, TGDM uses reconstruction-loss training without extra sampling tricks, replacing the unstable GAN and seesaw-style optimization paradigm.The figure caption describes this paradigm as stable while maintaining comparable results.Source: §1 · p. 2, Figure 1 · p. 2
  • Experiments are reported to demonstrate TGDM’s superiority over state-of-the-art methods across several face-manipulation tasks.The supplied introduction does not specify benchmark metrics or values.Source: §1 · p. 2

2 RELATED WORKS

Prior work spans structural, warping, audio-conditioned, and identity-transfer approaches, with limitations in identity preservation, modality-specific design, and training objectives. The cited diffusion work broadens conditional generation, while talking-face applications still face challenges in expressing pose and expression variation.

2.1 Talking Face Generation

  • Instruction-based and warping-based reenactment methods guide facial attributes through landmarks, maps, or motion fields, but may fail to explicitly capture source-target movement or preserve identity under extreme conditions.FOMM uses relative keypoints to predict flow fields; later work improves motion flow and warping accuracy.Source: §2.1 · p. 2
  • Audio-driven methods predict 2D points, 3D parameters, or radiance fields; emotion-conditioned and diffusion approaches exist, but modality-specific generators and limited expression and pose variation remain concerns.The section also describes prior diffusion methods for generalized talking-head synthesis.Source: §2.1 · p. 2

2.2 Face Swapping

  • Face-swapping methods range from 3D and GAN-based systems to StyleGAN and diffusion, yet identity transfer while preserving target attributes remains difficult under seesaw-style losses.The section presents reconstruction-loss training as the paper’s proposed response to this challenge.Source: §2.2 · pp. 2–3

2.3 Diffusion Model and Multimodal Generation

  • Diffusion models add noise and learn reverse denoising without discriminators, while conditional variants support guided generation; DDIM and latent diffusion address slow sampling.The paper places its multimodal geometry-guided face animation within this broader diffusion-generation context.Source: §2.3 · p. 3

3 PRELIMINARIES

The preliminaries introduce diffusion’s forward-noise and reverse-denoising processes, then describe 3DMM coefficients and rendering as a structured route from face parameters to image-space geometry. Rendered faces provide explicit geometry and detailed semantics for the later method.

3.1 Denoising Diffusion Probabilistic Models (DDPMs)

  • DDPMs define a forward Markov process that adds Gaussian noise to target data and a learned reverse process that recovers it; the resulting estimate supports pixel-level and perceptual training losses.The reverse transition is parameterized by a neural network, and sampling uses predicted noise.Source: §3.1 · p. 3

3.2 3D Morphable Models (3DMMs)

  • A 3DMM predicts 261 coefficients for identity (80), expression (64), texture (80), illumination (27), pose (6), and gaze (4), parameterizing 3D shape and albedo before differentiable projection to 2D.Its explainable, disentangled parameter space enables recombining corresponding factors for face manipulation.Source: §3.2 · p. 3

4 METHOD

The method represents multimodal driving conditions as 3DMM geometry and uses TGDM to transfer source texture to rendered geometry. It also applies TGDM to face swapping.

4.1 Overview

  • TGDM unifies multimodal talking-face generation through multi-conditional diffusion and extends the same framework to face swapping.Rendered faces projected from 3DMMs provide an intermediate structural representation for the geometry conditions.Source: §4.1 · p. 4

4.2 Multimodal-driven Geometry Condition

  • Image and video driving combine source appearance coefficients with driving motion coefficients; video uses adjacent-frame parameters in a radius-1 window to smooth motion.For audio driving, a non-autoregressive Transformer captures short- and long-term audio context for sequence-level expression regression.Source: §4.2 · p. 4
  • Text prompts encode emotion through CLIP, while EmoA2E combines audio cues, identity, emotion embeddings, and a learnable intensity token to predict expression.The intensity scalar is trained at values 1, 2, or 3 and may vary continuously from 1 to 3 at test time.Source: §4.2 · p. 4
  • EmoA2E training uses expression reconstruction and landmark losses, with landmarks selected from the original and modified 3DMM descriptors.The expression loss compares predicted and ground-truth expressions; the landmark loss measures similarity between their selected points.Source: §4.2 · p. 4

4.3 Texture-Geometry-aware Diffusion Model

  • TGDM reframes generation as target-oriented texture transfer, using diffusion to transfer source appearance to rendered geometry rather than rearranging source features.The model uses a UNet-based conditional denoiser.Source: §4.3 · p. 4, Figure 2 · p. 5
  • Texture Attention uses cross-attention to integrate source texture features with noise features, while rendered faces and modified coefficients condition geometry through concatenation and AdaIN.The modified coefficients add implicit geometry cues, including gaze direction not represented in the rendered face.Source: §4.3 · p. 5
  • TGDM training adds pixel-level reconstruction and perceptual constraints on the fully denoised face, using L2 and LPIPS losses.The perceptual loss uses a pretrained VGG16 network.Source: §4.3 · pp. 5–6

4.4 A Novel Face Swapping Paradigm Built on TGDM

  • For face swapping, TGDM reconstructs the source using identity and attribute conditions; inference combines source identity with target geometry to generate the swapped face.The scheme masks the source face, supplements identity cues with an identity embedding, and uses reconstruction loss during training.Source: Figure 4 · p. 6, §4.4 · p. 6

5.1 Datasets and Implementation Details

The experiments use VoxCeleb1 for talking-face generation, MEAD for emotional generation, and CelebAMask-HQ for face-swapping training with FaceForensics++ testing. The reported qualitative comparisons include challenging pose, occlusion, and face-size differences.

5.1 Datasets and Implementation Details

  • VoxCeleb1 provides 17,927 training and 491 testing videos after selecting high-resolution clips and cropping them to 256 × 256.The dataset contains over 20K videos; the selected clips are 720P.Source: §5.1 · p. 6
  • MEAD supplies eight emotion types and three intensity levels, with 36 front-view identities selected for training and the remainder for testing.CelebAMask-HQ supplies 30,000 images for face-swapping training, while FaceForensics++ is used for testing.Source: §5.1 · p. 6
  • The VoxCeleb1 qualitative comparisons highlight cases with substantial source–target differences in pose, occlusion, and face size.The figure directs attention to the area marked by a red arrow.Source: Figure 5 · p. 7

5.2 Face Reenactment

On VoxCeleb1, the method performs favorably on facial attributes, reconstruction, and image quality, while qualitative comparisons highlight robustness to pose, expression, and occlusion differences. Ablations support the roles of cross-attention and rendered-face geometry conditioning.

5.2.1 Comparison with Baselines

  • In challenging pose, face-size, expression, and occlusion cases, the method produces desired attributes and detailed expressions while preserving source identity and non-facial content.Compared methods exhibit over-smoothed textures, warping, pose inconsistency, or difficulty handling occluded regions.Source: §5.2.1 · pp. 7–8
  • The method is favorable on facial attributes, PSNR, LPIPS, and FID across same- and cross-identity VoxCeleb1 evaluations, while HifiHead has the lowest FID and strongest identity consistency but substantial pose error.The quantitative comparison is reported in Table 1; the text notes that HifiHead’s pose error is also apparent in the qualitative comparisons.Source: §5.2.1 · p. 8, Table 1 · p. 8

5.2.2 Ablation Study and Analysis

  • Image- and feature-level concatenation generate pose and expression but retain source appearance poorly, while geometry descriptors without the rendered face provide insufficient alignment.The reported ablation attributes higher quality to cross-attention and the rendered face as an explicit geometry condition, which also reduce training difficulty and speed convergence.Source: §5.2.2 · p. 8
  • TextAtt attention maps show that sampled head, face, and background locations attend to geometrically and semantically similar source regions.The authors connect this correspondence-based design with photo-realistic, identity-consistent texture transfer.Source: Figure 7 · p. 8, §5.2.2 · p. 9

5.3 Talking Face Generation

For audio-driven talking-face generation, the method combines accurate motion control and temporal coherence with identity consistency and image quality. Its emotion-conditioned setting supports varied emotion styles and intensity levels, with a reported synchronization trade-off.

5.3.1 Comparison with Baselines

  • Qualitative comparisons report faithful pose, audio-driven mouth movement, and source texture, while other methods show issues including pose changes, blending artifacts, or limited emotion generation.Some compared methods require aligned inputs, which the text says can destroy facial structure and contribute to pose misalignment.Source: §5.3.1 · p. 9
  • The method yields the best LMD, Sync, ID-C, and FID among compared emotion-free talking-face methods in the VoxCeleb1 reconstruction evaluation.The comparison is reported in the top part of Table 3.Source: §5.3.1 · p. 9, Table 3 · p. 9
  • On MEAD, emotion-conditioned results outperform most metrics except FID; compared with emotion-free generation, they improve mouth-shape accuracy but have lower synchronization.The authors attribute the mouth-shape result to MEAD’s limited corpus and the synchronization result to irregular talking rhythm introduced by emotion.Source: Table 3 · p. 9, §5.3.1 · p. 10

5.3.2 Ablation Study and Analysis

  • Transformer-based emotion encoding achieves LMD 2.36 versus 3.54 for MLPs and 2.47 for GRUs, while CLIP similarity is 0.621 versus 0.430 for GPT2.The CLIP–GPT2 comparison measures cosine similarity between generated sequences and corresponding text prompts.Source: §5.3.2 · p. 10
  • The method flexibly manipulates compound and unseen emotion styles, including Hatred, and controls expression continuously by increasing intensity from 1 to 2.5, producing accurate changes.Visual changes include the mouth and eyes.Source: §5.3.2 · p. 10

5.4 Expanded Application of Face Swapping

In face swapping, the method is reported to balance source-identity consistency with preservation of target attributes. Ablations examine mask size and the rendered face condition used in the transfer.

5.4.1 Comparison with Baselines

  • Compared with DiffFace, the results show stronger identity consistency and attribute preservation but some reduction in visual quality.The authors say DiffFace produces clear but inconsistent textures, whereas their results are more faithful to the target.Source: §5.4.1 · p. 10
  • The method is reported to perform better when considering both source identity consistency and target attribute preservation in FaceForensics++ face swapping.The quantitative comparison is summarized in Table 4; qualitative results describe improved source-like face shape and preservation of target hair and backgrounds.Source: §5.4.1 · pp. 10–11, Table 4 · p. 11

5.4.2 Ablation Study and Analysis

  • Normal and Small masks show no apparent identity or attribute difference, but Small gives a more realistic mouth; Dilated masks create contour artifacts and image degradation.The authors select the Small mask based on these observations.Source: §5.4.2 · p. 11, Figure 12 · p. 11
  • Without the rendered face condition, swapped-result colors tend to resemble the source rather than the target.This observation is presented as support for using the rendered face as a condition.Source: §5.4.2 · p. 11

6 LIMITATIONS AND FUTURE WORKS

TGDM’s single-image generation leaves temporal inconsistency unresolved, while diffusion-model runtime and supported resolution constrain practical use. Frame-by-frame reuse can smooth transitions but risks propagating errors after sudden movements.

6 LIMITATIONS AND FUTURE WORKS

  • Single-image generation introduces temporal inconsistency, and frame-by-frame reuse can propagate errors after sudden movements, degrading all subsequent frames.Reusing the synthesized image as the next source can smooth adjacent frames, but the authors leave temporal coherence for future work.Source: §6 · p. 11
  • 45 ms per face on one V100 under T = 1000 DDPM is described as unacceptable for real applications.The authors identify DDIM and LDMs as approaches that have alleviated diffusion-model efficiency problems.Source: §6 · p. 11
  • For efficiency, the model supports only 256 × 256 image generation.The authors hope to develop an intuitive design for efficient high-resolution face generation.Source: §6 · p. 11
  • The authors did not train the model longer because of diffusion’s high consumption.Source: §6 · p. 11

7 CONCLUSION

The paper proposes TGDM for multimodal-driven talking face generation, using text prompts for emotion control and target-oriented texture transfer. It also applies TGDM to face swapping and reports superiority across face manipulation tasks.

7 CONCLUSION

  • TGDM treats talking face generation as target-oriented texture transfer, preserving source textures and appearance details while retaining explicit structure and avoiding complex texture deformations.This formulation allows all driving modalities to share one generator.Source: §7 · p. 11
  • Text prompts represent emotion using CLIP’s rich semantics, supporting flexible and generalized emotion control.Source: §7 · p. 11
  • TGDM also supports face swapping through reconstruction-based training, avoiding seesaw-style optimization during inference.Source: §7 · p. 11
  • Extensive results demonstrate the proposed pipeline’s superiority across various face manipulation tasks.Source: §7 · p. 11

Related papers

Loading 2305.02594v2…