PaperlayerView exact revision on arXiv

Source-linked AI summary

MarioNETte: Few-shot Face Reenactment Preserving Identity of Unseen Targets

Sungjoo Ha, Martin Kersner, Beomsu Kim, Seokjun Seo, Dongyoung Kim

arXiv:1911.08139v1cs.CV

TL;DR

Few-shot face reenactment struggles to preserve unseen target identities when driver and target characteristics differ, producing defective outputs. MarioNETte combines image attention, target feature alignment, and landmark transformation to address this mismatch, and experiments with user studies report that it outperforms state-of-the-art methods. Large pose differences remain a reported source of failure, motivating future improvement of landmark disentanglement.

  • Problem

    Few-shot face reenactment methods struggle to preserve target identity, and the problem worsens when driver and target identities diverge.

  • Method

    MarioNETte combines image attention, target feature alignment, and unsupervised landmark transformation to transfer target information and adapt identity-related geometry.

  • Results

    MarioNETte outperforms state-of-the-art methods in evaluations and user studies covering matching and differing target-driver identities.

  • Takeaways & Limitations

    The framework preserves target identity without additional fine-tuning, including when target and driver facial characteristics differ widely.

  • Takeaways & Limitations

    Large pose differences between driver and target remain a main source of MarioNETte+LT failures, and future work targets improved landmark disentanglement.

Abstract

from arXiv · show

When there is a mismatch between the target identity and the driver identity, face reenactment suffers severe degradation in the quality of the result, especially in a few-shot setting. The identity preservation problem, where the model loses the detailed information of the target leading to a defective output, is the most common failure mode. The problem has several potential sources such as the identity of the driver leaking due to the identity mismatch, or dealing with unseen large poses. To overcome such problems, we introduce components that address the mentioned problem: image attention block, target feature alignment, and landmark transformer. Through attending and warping the relevant features, the proposed architecture, called MarioNETte, produces high-quality reenactments of unseen identities in a few-shot setting. In addition, the landmark transformer dramatically alleviates the identity preservation problem by isolating the expression geometry through landmark disentanglement. Comprehensive experiments are performed to verify that the proposed framework can generate highly realistic faces, outperforming all other baselines, even under a significant mismatch of facial characteristics between the target and the driver.

Introduction

MarioNETte addresses identity preservation failures in few-shot face reenactment, especially when target and driver identities differ. It combines feature-based components with landmark transformation and is reported to outperform state-of-the-art methods across matching and mismatched identities.

  • Problem: Few-shot reenactment methods can lose target identity, with failures worsening as driver and target identities diverge.The resulting face may become defective or resemble the driver.
  • Failure modes: Previous approaches exhibit driver-shape interference, loss of target details, and warping defects at large poses.These failure modes are illustrated in Figure 1.
  • Proposed framework: Image attention and target feature alignment inject relevant target information and use feature-level warping to improve reenactment under different identities.The attention block selects relevant target feature-map positions, while alignment performs multiple feature-level warping operations.
  • Proposed framework: Landmark transformation adapts driver landmarks to the target in an unsupervised manner, mitigating identity preservation problems without additional labeled data.This component is intended to cope with varying facial characteristics across people.
  • Evaluation: Experiments and user studies compare methods on VoxCeleb1 and CelebV, with the proposed method outperforming state-of-the-art baselines.The evaluation covers both coinciding and differing target-driver identities.

MarioNETte Architecture

MarioNETte combines driver pose and expression encoding with target feature extraction, attention-based blending, feature alignment, and decoding. Its attention mechanism selects spatial target information across multiple images, while alignment normalizes target pose and adapts features to the driver pose.

  • Overall architecture: The conditional generator receives a driver image and target images, while a discriminator predicts whether generated images are real.The generator is organized around preprocessing, encoding, blending, and decoding components.
  • Overall architecture: The preprocessor extracts and normalizes 3D facial landmarks for driver and target inputs, including the landmark transformer.The landmarks are rendered into landmark images for subsequent processing.
  • Feature encoding: The driver encoder extracts pose and expression information, while the target encoder extracts style information and warped target feature maps.The target encoder uses a U-Net architecture.
  • Feature blending: The blender combines driver and multiple target feature maps, using the image attention block as its basic building block.The resulting mixed feature map is passed to the decoder.
  • Image attention block: Image attention uses driver features as queries and target features as memory to attend to relevant positions across multiple target maps.Positional encodings represent feature-map coordinates, and the output is reshaped into a spatial feature map.
  • Target feature alignment: Target feature alignment warps target features in two stages: target pose normalization followed by driver pose adaptation.The decoder first normalizes target features and then estimates a driver-conditioned flow to align them before residual upsampling.

Landmark Transformer

The landmark transformer addresses large structural differences between target and driver landmarks by transferring driver expression to an arbitrary target identity without labeled paired data. It disentangles identity and expression geometry using normalized landmarks, expression bases, and a learned coefficient predictor.

  • Motivation: Large structural differences between facial landmarks can severely degrade reenactment quality, while paired landmark data and identity-specific transformations are unsuitable for unseen identities.The landmark transformer is introduced to address this few-shot setting without requiring labeled data.
  • Landmark decomposition: Normalized 3D landmarks are modeled as the sum of mean facial geometry, identity geometry, and frame-specific expression geometry.Landmarks are normalized for scale, translation, and rotation before decomposition.
  • Landmark decomposition: The desired transformed landmark combines the target identity geometry with the driver expression geometry, but this decomposition is difficult for unseen targets in few-shot settings.The transformer therefore targets expression transfer while preserving target-specific structure.
  • Landmark disentanglement: Expression landmarks are separated into semantic face groups, and PCA extracts linear expression bases from training data.The bases cover groups such as the mouth, nose, and eyes.
  • Landmark disentanglement: A landmark disentangler estimates expression-basis coefficients from an image and its normalized landmark, enabling identity and expression geometry to be computed.The network M predicts coefficients using image features and landmarks, while λ_exp controls predicted-expression intensity.
  • Inference: During inference, target and driver landmarks are transformed, averaged across multiple target images when available, denormalized, and rasterized for the generator.The resulting landmark representation is formatted for consumption by the face generator.

Experimental Setup

The experiments train MarioNETte and baselines on VoxCeleb1, then evaluate self-reenactment and different-identity reenactment using image-quality, identity-preservation, pose, and expression metrics. CelebV supplies the different-identity evaluation, with qualitative comparisons shown for the one-shot setting.

  • Datasets: Models are trained on VoxCeleb1, which contains 256 × 256 videos from 1,251 identities, and evaluated on VoxCeleb1 and CelebV test data.VoxCeleb1 supports self-reenactment evaluation, while CelebV evaluates reenactment under a different identity.
  • Baselines: MarioNETte variants with and without the landmark transformer are compared with state-of-the-art few-shot face-reenactment models.The listed baselines include X2Face, Monkey-Net, and NeuralHead-FF.
  • Evaluation: Qualitative results compare generated images from MarioNETte and baselines for different-identity reenactment on CelebV in the one-shot setting.The comparison is presented in Figure 6 alongside the corresponding evaluation results.
  • Metrics: SSIM, PSNR, masked-SSIM, and masked-PSNR measure low-level similarity between generated and ground-truth images, including facial-region-restricted variants.The masked metrics restrict measurements to the facial region.
  • Metrics: CSIM evaluates identity preservation when ground-truth images are unavailable for different-identity driving, while PRMSE and AUCON evaluate driver pose and expression reproduction.Pose angles and facial action-unit values are computed with OpenFace.

Experimental Results

MarioNETte is evaluated in self-reenactment, different-identity reenactment, user studies, and ablations. It generally improves identity preservation and realism, while landmark transformation trades some expression-transfer performance for stronger identity preservation.

  • Self-reenactment: MarioNETte surpasses other models on every metric under few-shot self-reenactment and all but PSNR under one-shot self-reenactment.It achieves the best M-PSNR, indicating stronger performance on the facial region than baselines.
  • Different-identity reenactment: MarioNETte and MarioNETte+LT preserve target identity adequately and outperform other models in CSIM when reenacting different identities.The method alleviates identity-preservation problems whether driver and target identities coincide or differ.
  • Different-identity reenactment: MarioNETte+LT significantly boosts identity preservation but slightly decreases PRMSE and AUCON.The decrease may reflect insufficiently diverse PCA bases for expression disentanglement, especially when identity and expression must be disentangled in one-shot settings.
  • User study: Human evaluations found the proposed models preferred over existing methods, with MarioNETte+LT achieving nearly twice MarioNETte’s realism score in few-shot settings.MarioNETte was slightly preferred overall because the landmark transformer can slightly degrade expression transfer.
  • Ablation test: Ablations show that attention improves identity preservation, alignment alone struggles with unseen poses and expressions, and combining both components outperforms +Alignment on every metric.The alignment-only failure is associated with warping low-level features from large-pose inputs and aggregating targets with diverse poses.

Related Works

Earlier face-reenactment work uses explicit 3D modeling, image warping, style-spatial blending, attention, deformation, and GAN-based methods. These approaches combine target appearance information with driver motion or spatial information through different representations and operations.

  • Classical methods: Classical methods compute 3D face-model parameters or estimated flow from 3D models and sparse landmarks, then modify or blend target imagery.These approaches include explicit 3D modeling and image warping.
  • Recent architectures: Recent architectures blend target style information with driver spatial information using AdaIN, attention, deformation, or GAN-based operations.These mechanisms have also been applied to few-shot face reenactment settings.

Conclusions

MarioNETte combines image attention, target feature alignment, and a landmark transformer to address identity mismatch in few-shot face reenactment. It requires no additional fine-tuning for identity adaptation, while improving landmark disentanglement remains future work.

  • Contributions: Image attention, target feature alignment, and the landmark transformer address identity mismatch caused by using landmarks from a different person.The framework is designed for few-shot face reenactment.
  • Contributions: MarioNETte requires no additional fine-tuning phase for identity adaptation, increasing its usefulness for in-the-wild deployment.Experiments, including human evaluation, support the proposed method’s performance.
  • Future work: Future work should improve the landmark transformer’s landmark disentanglement to make reenactment more convincing.This is identified as an avenue for improving the method.

MarioNETte Architecture Details

MarioNETte combines landmark-conditioned target and driver encoding with attention-based feature blending, feature warping, and adversarial generation to produce reenacted images while retaining target information.

  • Landmark representation: MarioNETte extracts 3D facial keypoints, rasterizes them into colored 2D landmark images, and uses these representations as inputs to the reenactment framework.The rasterizer groups projected landmarks into eight facial categories and draws colored lines for each group.
  • Adversarial training: A projection discriminator evaluates synthesized images conditionally on rasterized landmarks and identity, while generator training combines adversarial, perceptual, and feature-matching losses.The discriminator produces scores over multiple patches, and perceptual losses use both ImageNet and face-recognition features.
  • Target encoding: The target encoder processes each target image and its landmark image into a target feature map and normalized multiscale feature maps for later alignment.The most downsampled feature map becomes the encoded target feature, while the remaining maps are transformed for warping.
  • Feature blending: The driver encoder produces a driver feature map, while the blender combines driver positional information with target style features to form a mixed feature map.Three stacked image attention blocks build the blender.
  • Decoding and alignment: The decoder applies warp-alignment and residual upsampling blocks to generate the reenacted image from the mixed feature map and warped target features.The flow map has two channels representing horizontal and vertical warping directions, and bilinear sampling provides differentiable feature warping.

Landmark Transformer Details

The landmark transformer addresses few-shot landmark disentanglement by estimating expression geometry from unlabeled training data and transferring driver expressions to unseen target identities.

  • Motivation: Landmark decomposition separates identity and expression geometry, but direct decomposition is unavailable for unseen identities with only one or a few insufficiently diverse frames.The expression mean can be zero or near zero when the available expressions are not diverse enough.
  • Landmark disentanglement: The landmark disentangler enables the decomposition needed for one-shot and few-shot settings despite limited target frames.It is introduced specifically to overcome the failure of direct decomposition for unseen identities.
  • Expression basis: Expression bases are computed by applying group-wise PCA to expression geometry from VoxCeleb1, yielding 48 bases across facial landmark groups.The PCA dimensions are 8, 8, 8, 16, and 8 for the listed groups.

Additional Ablation Tests

Additional ablations show that MarioNETte's relative performance depends on dataset and identity setting, while its components contribute distinct qualitative benefits and large pose differences remain problematic.

  • Cross-setting comparison: MarioNETte outperforms NeuralHead-FF on PRMSE and AUCON for self-reenactment on VoxCeleb1, but NeuralHead-FF performs better for different-identity reenactment on CelebV.The reported reversal is examined through an ablation study.
  • Self-reenactment ablation: Under VoxCeleb1 self-reenactment, +Alignment and MarioNETte achieve better PRMSE and AUCON than AdaIN.The authors attribute the phenomenon to training-dataset characteristics and different model inductive biases.
  • Identity preservation: For different-identity reenactment on CelebV, +Attention maintains target characteristics where AdaIN fails to generate an image resembling the target identity.The comparison is shown for both one-shot and few-shot settings.
  • Failure cases: Large pose differences between driver and target appear to be the main reason for MarioNETte+LT failures in one-shot, different-identity VoxCeleb1 reenactment.These failures are reported as qualitative cases in Figure 18.
  • Component effects: The alignment module adds fine-grained details, whereas MarioNETte produces more natural few-shot images and +Alignment struggles with diverse target poses and expressions.The comparison separates detail enhancement from robustness to multiple diverse target images.

Inference Time

Inference is evaluated for 256 × 256 generation with one or eight target images, separating target encoding from driver generation and showing different scaling behavior across components.

  • Measurement setup: Inference measurements average 300 runs per setting on an Nvidia Titan Xp using PyTorch 1.0.1.post2 for K ∈{1, 8} target images.The evaluation reports latency while generating 256 × 256 images.
  • Pipeline breakdown: The inference pipeline is divided into target encoding and driver generation because target encoding features are computed once at the beginning of reenactment-video generation.The cached quantities are the encoded target feature and warped target features.
  • Scaling behavior: Target-side components scale sublinearly with the number of target images under batched inference, whereas the sequential 3D landmark detector scales linearly.This contrast applies to components such as the target encoder and target landmark transformer versus landmark detection.

Additional Examples of Generated Images

The paper provides qualitative comparisons across VoxCeleb1 and CelebV, covering self-reenactment, identity-mismatched reenactment, ablations, and failure cases in one-shot and few-shot settings. It also reports inference-speed measurements for individual components and the full model.

  • Additional qualitative results compare baseline methods and proposed models on VoxCeleb1 and CelebV in one-shot and few-shot settings.Few-shot evaluations generally use eight target images, although only one target image is displayed because of space constraints; Monkey-Net uses one image by design.
  • VoxCeleb1: Figures 11–14 cover self-reenactment and identity-mismatched reenactment on VoxCeleb1 across one-shot and few-shot settings.Figures 11 and 12 address self-reenactment, while Figures 13 and 14 address mismatched driver and target identities.
  • CelebV: Figures 15–17 present CelebV comparisons for one-shot and few-shot self-reenactment and few-shot reenactment of different identities.The CelebV examples include comparisons among various methods and a separate few-shot different-identity setting.
  • Inference speed: Tables 6 and 7 report inference speed for model components and for the full model generating a single image from K target images.
  • Ablation results: Figures 9 and 10 show qualitative ablation results for one-shot and few-shot reenactment under different identity settings on CelebV.
  • Failure cases: Figure 18 documents failure cases produced by MarioNETte+LT during one-shot reenactment with different identities on VoxCeleb1.
Loading 1911.08139v1…