PaperlayerView exact revision on arXiv

Source-linked AI summary

One-shot Face Reenactment

Yunxuan Zhang, Siwei Zhang, Yue He, Cheng Li, Chen Change Loy, Ziwei Liu

arXiv:1908.03251v1cs.CVeess.IV

TL;DR

One-shot face reenactment must transfer source pose and expression while preserving target identity, but existing methods usually need multiple target images or long target videos. This paper disentangles target appearance from source shape, composes them with a shared decoder, and adds FusionNet to combine synthesis with warping. The resulting system reports superior transfer fidelity and identity preservation to alternatives, while one target image per subject achieves results competitive with methods using target-image sets.

  • Problem

    Existing face reenactment methods rely on multiple target faces or long target videos, whereas this work targets realistic pose and expression transfer from one target image.

    Source: §1 · pp. 1–2

  • Method

    The framework encodes target appearance and source shape separately, composes them with a shared decoder, and uses FusionNet to combine learned synthesis with traditional warping.

    Source: §1 · pp. 2–3

  • Results

    The system reports superior transfer fidelity and identity preservation to alternatives, with one target image per subject achieving results competitive with methods using target-image sets.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    One-shot training and testing provide a practical face reenactment framework for scenarios where only one target image is available.

    Source: §1 · p. 3

    GANimation is supervised by action-unit labels, unlike the paper’s training setting.

    Source: §4.3 · p. 9

Abstract

from arXiv · show

To enable realistic shape (e.g. pose and expression) transfer, existing face reenactment methods rely on a set of target faces for learning subject-specific traits. However, in real-world scenario end-users often only have one target face at hand, rendering existing methods inapplicable. In this work, we bridge this gap by proposing a novel one-shot face reenactment learning framework. Our key insight is that the one-shot learner should be able to disentangle and compose appearance and shape information for effective modeling. Specifically, the target face appearance and the source face shape are first projected into latent spaces with their corresponding encoders. Then these two latent spaces are associated by learning a shared decoder that aggregates multi-level features to produce the final reenactment results. To further improve the synthesizing quality on mustache and hair regions, we additionally propose FusionNet which combines the strengths of our learned decoder and the traditional warping method. Extensive experiments show that our one-shot face reenactment system achieves superior transfer fidelity as well as identity preserving capability than alternatives. More remarkably, our approach trained with only one target image per subject achieves competitive results to those using a set of target images, demonstrating the practical merit of this work. Code, models and an additional set of reenacted faces have been publicly released at the project page.

1 Introduction

The paper frames one-shot face reenactment as transferring source pose and expression while preserving target identity from only one target image. It addresses this gap by disentangling appearance and shape, composing them with a shared decoder, and adding FusionNet for facial texture.

  • Existing reenactment methods require many target faces or long target videos, limiting realistic subject-specific transfer when only one target image is available.The one-shot setting also makes arbitrary-view appearance synthesis, unseen-expression modeling, and identity preservation difficult.Source: §1 · p. 2
  • The proposed learner separates target appearance from source shape and combines them through a decoder D : B×A →X for one-shot reenactment.The appearance representation must preserve texture while excluding pose and expression information.Source: §1 · pp. 2–3
  • Multi-level appearance features, SPADE modules, and linked reconstruction and reenactment branches are used to preserve identity, texture, and spatial shape information.The appearance encoder and reenactment decoder are trained jointly, while feature maps from the appearance decoder are inserted into multiple decoder layers.Source: §1 · p. 3
  • FusionNet combines the learned synthesis result with traditional warping to improve facial texture and mustache generation.The approach is illustrated as reenacting arbitrary pose guides from one reference image using one model trained with one-shot data.Source: Figure 1 · p. 2, §1 · p. 3
  • Experiments report that one-shot training outperforms single-image face-generation methods and is competitive with subject-specific methods using multiple target images.The authors present this as practical evidence for one-shot training and testing.Source: §1 · p. 3

2 Related Work

Prior face reenactment methods either fit subject-specific models from substantial data or manipulate limited facial attributes from one image. These approaches struggle with one-shot reenactment involving large pose changes, complex expressions, and realistic facial details.

  • Model-based and feed-forward reenactment methods typically require substantial subject-specific data and separate training for each target face.Their pipelines capture movement, fit a parametric or 3D model, and render or generate the final video.Source: §2 · pp. 3–4
  • SPADE is identified as equivalent to Spatial Feature Transform with normalization layers.Source: §2 · p. 3
  • One-shot facial manipulation methods support single-image input but mainly alter facial components and struggle with large shape or pose changes.Other one-shot generators can suffer identity shift or coarse facial details when used for face images.Source: §2 · p. 4
  • Prior work does not enable one-shot face reenactment with an explicit continuous pose representation because of large shape changes, complex pose and expression, and realistic-detail requirements.Source: §2 · p. 4

3 Approach

The approach encodes target appearance and source shape separately, then progressively composes them into a reenacted face. It augments this decoder with multi-scale spatial conditioning, FusionNet blending, and jointly optimized reconstruction, perceptual, adversarial, and identity losses.

3 Approach

  • The model computes D(F(x_t),E(x_s)) by encoding target appearance and source pose into separate latent spaces before decoding their combination.F maps faces to appearance representations, E maps faces to shape representations, and D returns the reenacted face.Source: §3 · p. 4

3.1 Disentangle-and-Compose Framework

  • The disentangle-and-compose framework uses a pretrained shape encoder, a learned appearance auto-encoder, multi-scale feature concatenation, and SPADE blocks to preserve identity details and source spatial semantics.The appearance encoder is trained to reconstruct the target while randomized pose contours encourage shape independence; the decoder is trained jointly with it.Source: §3.1 · pp. 4–6, Figure 3 · p. 5

3.2 FusionNet

  • FusionNet generates a mask that fuses the proposed model’s result with a traditional warped result, combining large-pose and exaggerated-action handling with improved local facial texture.Warping is especially useful for facial details such as mustache and wrinkles.Source: §3.2 · p. 6, Figure 4 · p. 6

3.3 Learning

  • The learning objective jointly balances reconstruction, perceptual, GAN, and identity losses, with λ = 25, α_r = 25, and α_p,α_g,α_i = 1.Source: §3.3 · p. 6

4 Experiments

The experiments compare one-shot face reenactment with single-image and target-specific alternatives, using qualitative and quantitative evaluations of expression, pose, and identity preservation. The framework preserves identity while transferring expression and head pose, remains competitive with limited target data, and benefits from FusionNet and feature concatenation.

4.1 Comparison with Single-Image Methods

  • The framework simultaneously modifies inner-face expression and whole-head pose while preserving identity, unlike baselines that exhibit frozen pose, identity shift, or blurred faces.The qualitative comparison covers GANimation, PG2, gauGAN, VU-Net, and the proposed model.Source: §4.1 · pp. 7–8, Figure 5 · p. 8

4.2 Comparison with Target-Specific Methods

  • With only one target image and no extra training, the framework achieves competitive reenactment results where ReenactGAN fails as target data decreases.Target-specific methods require substantial target data, computation, and collection effort.Source: §4.2 · p. 8, Figure 6 · p. 9

4.3 Quantitative Results

  • Across identical-source, cross-source, and in-the-wild data, the proposed model shows strong AU and pose consistency and overall high performance, while preserving identity competitively with GANimation.Table 1 evaluates AU consistency, head-pose consistency, and identity preservation; higher is better for AU and identity, while lower is better for pose error.Source: §4.3 · pp. 9–10, Table 1 · p. 9
  • FusionNet significantly improves identity preservation but causes a small AU-consistency drop because it prioritizes face quality over expression.PG2 has the best AU consistency because it is trained with paired data that provides stronger expression supervision.Source: §4.3 · p. 10
  • Concatenating appearance-decoder and spatially adaptive decoder features significantly improves identity preservation, whereas an overly large λ overfits appearance reconstruction.The ablation evaluates identity preservation under different appearance-reconstruction settings.Source: §4.3 · p. 10, Table 2 · p. 10
  • Identity preservation improves when the model receives more target images in 3-shot and 5-shot experiments.The few-shot dataset contains 32 identities, each with seven images spanning different expressions and poses.Source: §4.3 · p. 10, Table 3 · p. 10

5 Conclusion

The paper presents one-shot face reenactment from a single target image by disentangling appearance and expression, then composing them for synthesis. FusionNet combines learned synthesis with traditional warping to improve texture and mustache generation, while experiments show competitive results against methods using multiple target images.

5 Conclusion

  • The model transfers a source face’s expression to a target face using only one target image.It independently models appearance and expression before composing them into reenacted results.Source: §5 · p. 10
  • FusionNet combines generated and traditionally warped results to improve texture and mustache quality.The design complements the disentangled appearance-and-expression framework.Source: §5 · p. 10
  • Training with one target image per subject achieves competitive results with approaches using a set of target images.The conclusion identifies this as the practical merit of the work.Source: §5 · p. 10
Loading 1908.03251v1…