Source-linked AI summary
Neural Head Reenactment with Latent Pose Descriptors
Egor Burkov, Igor Pasechnik, Artur Grigorev, Victor Lempitsky
TL;DR
The paper addresses how to represent pose for neural head reenactment without relying on person-specific landmarks or explicit pose supervision. It learns latent pose and identity descriptors from reconstruction losses, then shows realistic same-person and cross-person reenactment while preserving target identity. The descriptors also support pose-related tasks, although subtle mimics such as gaze remain difficult.
Problem
Landmark-based pose representations require annotation, can omit pose aspects such as gaze, and retain person-specific information that limits cross-person reenactment.
Method
The system learns low-dimensional person-agnostic pose descriptors and person-specific pose-independent descriptors using reconstruction losses, pose augmentation, limited pose-encoder capacity, and foreground segmentation.
Results
The learned descriptors enable realistic same-person and cross-person reenactment with limited identity bleeding, and outperform previous unsupervised descriptors in pose-based retrieval and cross-person reenactment.
Takeaways & Limitations
Reconstruction-based learning on large video datasets can disentangle pose from identity sufficiently for cross-person reenactment and related pose tasks without explicit pose supervision.
Takeaways & Limitations
The system has difficulty capturing some subtle mimics, especially gaze direction, and comparison with a recent unsupervised-keypoint method is left for future work.
Abstract
from arXiv · showhide
We propose a neural head reenactment system, which is driven by a latent pose representation and is capable of predicting the foreground segmentation alongside the RGB image. The latent pose representation is learned as a part of the entire reenactment system, and the learning process is based solely on image reconstruction losses. We show that despite its simplicity, with a large and diverse enough training dataset, such learning successfully decomposes pose from identity. The resulting system can then reproduce mimics of the driving person and, furthermore, can perform cross-person reenactment. Additionally, we show that the learned descriptors are useful for other pose-related tasks, such as keypoint prediction and pose-based retrieval.
1. Introduction
The paper replaces landmark-based pose representations with learned latent descriptors for neural head reenactment. Using reconstruction losses over large unlabeled video data, it aims to disentangle pose from identity and support cross-person reenactment.
- Motivation: Landmark representations require costly annotation, can omit important pose aspects such as gaze, and may be ambiguous or temporally unstable.They also contain person-specific geometry, which is undesirable when driving a target with different head geometry.
- Approach: The method learns low-dimensional person-agnostic pose descriptors alongside person-specific pose-independent descriptors from reconstruction losses on video frames.Background segmentation prevents clutter and its frame-to-frame changes from affecting the learned descriptors.
- Results: A simple framework using multiple random frames from each video learns descriptors that support reenactment, including cross-person reenactment.The learned pose descriptors are also analyzed for landmark prediction and pose-based retrieval.
- Results: The system demonstrates realistic reenactment while preserving the target identity when arbitrary people provide the driving pose.The figure illustrates this capability with talking-head subjects including Mona Lisa.
2. Related work
Prior reenactment systems use landmarks, action units, warping descriptors, or 3D models, each retaining annotation, person-specific, or data requirements. The paper situates latent pose descriptors as an alternative for person-agnostic cross-person transfer.
- Reenactment scope: Face and head reenactment methods differ in whether they modify only facial regions or model extended regions including clothing, neck, and upper garments.The paper focuses on the broader head-reenactment setting.
- Pose representations: Landmarks and action units provide common pose representations, but action units still require manual annotation and supervised learning.X2Face instead learns latent vectors that predict warping fields.
- Pose representations: 3D morphable models typically rely on landmark detection or require 3D-scan datasets for pose-identity disentanglement.These requirements inherit landmark deficiencies or impose additional data demands.
- Transferability: Unsupervised keypoints, dense descriptors, and person-specific latent descriptors generally do not establish person-to-person transfer in the cited approaches.The paper notes that transfer of codec-avatar descriptors was not considered.
- Disentangled representations: Related disentanglement methods learn latent pose or shape descriptors, content-style factors, or factorized distributions using video datasets, adversarial losses, cycle consistency, or direct fitting.These approaches provide broader context for the paper’s reconstruction-based disentanglement strategy.
3. Method
The system extends a neural reenactment model with segmentation prediction and latent pose vectors. It uses multiple frames to encode identity, a constrained augmented pose source, and a generator trained to reconstruct the held-out foreground image and mask.
- Model and data: The model adds foreground-segmentation prediction and replaces keypoint inputs with latent pose vectors in a prior reenactment architecture.Training uses VoxCeleb2 video sequences with meta-learning followed by identity-specific fine-tuning.
- Training pipeline: Each episode samples K + 1 frames; a high-capacity identity encoder averages K embeddings, while a lower-capacity pose encoder processes the held-out frame.The identity embedding represents pose-independent information such as lighting and clothing.
- Training pipeline: Pose augmentation preserves the source pose while altering identity-related appearance through scaling, blur, sharpening, contrast changes, or JPEG compression.This augmentation helps prevent the pose encoder from carrying identity information.
- Generator and losses: The generator receives identity and pose embeddings through AdaIN and predicts both a 3-channel RGB image and a 1-channel foreground mask.Reconstruction matches the masked target image and segmentation map using image losses and Dice loss.
- Inference: Given a new identity embedding and a pose vector from the same or another person, the system produces the target person’s reenacted image and foreground mask.Fine-tuning adapts the model to new identities while keeping the estimated identity embedding fixed.
- Results: Cross-person reenactment preserves the target identity with limited identity bleeding, and the learned disentanglement is stronger than in related approaches.The paper evaluates this effect through reenactment experiments and supplementary ablations of capacity, augmentation, segmentation, and pose-vector dimensionality.
- Disentanglement: Low pose-encoder capacity, pose augmentation, and background segmentation together yield pose-identity disentanglement without requiring additional adversarial or cycle-consistency losses.The authors report that extra loss terms did not further improve disentanglement.
- Implementation details: Training uses roughly seven million images from about 100,000 VoxCeleb2 videos, with K = 8 identity frames reconstructing a ninth frame.This large video collection supplies the diversity used for learning the descriptors.
4. Results
The evaluation compares learned pose descriptors and reenactment systems across retrieval, keypoint prediction, and cross-person reenactment. The full system achieves strong pose matching, preserves target identity while reproducing driver expressions, supports smooth pose interpolation and temporally smooth outputs, and outperforms or compares favorably with the evaluated alternatives.
- Evaluation setup: The evaluation covers auxiliary pose-descriptor tasks, cross-person reenactment, qualitative comparisons, interpolation, temporal smoothness, and component ablations.The compared descriptors and reenactment systems include the proposed model, X2Face, FAb-Net, 3DMM, and FSTH variants.
- Pose-based retrieval: The proposed latent pose embeddings group Multi-PIE images by emotion better than other facial-expression descriptors on top-10 and top-20 retrieval, while matching or exceeding alternatives on top-50 and top-100.The evaluation uses Multi-PIE images from camera orientations restricted to near-frontal and half-profile views; rigid pose is ignored for emotion retrieval.
- Keypoint prediction: The proposed descriptors are evaluated for keypoint prediction on the MAFL benchmark using both pose and identity embeddings as inputs, although keypoint prediction is not a target application.The paper notes that keypoints contain person-specific information.
- Cross-person reenactment: The full model is strictly better than all compared systems except FSTH, which is slightly better on one metric but much worse on the other and uses an external keypoint detector.The metrics measure pose representation and reference-identity preservation; lower-left performance is better, with both ideal metric values equal to 0.
- Qualitative reenactment: The full system accurately reproduces the driver’s facial expressions while preserving the target identity, whereas 3DMM+ has a small identity gap but weaker subtle-expression rendering.FSTH relies heavily on driver facial proportions, while FSTH+ still shows identity bleeding and expression errors; X2Face+ and FAb-Net+ show some identity mismatch.
- Temporal and pose-space behavior: Pose-space interpolation produces smooth pose changes, and temporally smooth reenactment is achieved without smoothing extracted pose when bounding-box detection is temporally smooth.Keypoint-driven FSTH and FSTH+ systems require substantial keypoint smoothing to achieve temporal smoothness.
5. Discussion
The system learns latent pose descriptors without explicit pose supervision and achieves realistic reenactment, including cross-person transfer with limited identity bleeding. Capacity constraints aid pose/identity disentanglement but can reduce fidelity for subtle mimics such as gaze.
- The system uses latent pose descriptors learned purely from reconstruction losses, with segmentation masks providing the only weak supervision.
- Its learned descriptors outperform previous unsupervised descriptors in pose-based retrieval and cross-person reenactment.
- Limiting pose-extractor capacity is sufficient for pose/identity disentanglement.
- Cross-person reenactment better preserves the target identity while transferring mimics from the driver person.
- The system has difficulty capturing subtle mimics, especially gaze direction, although it exceeds keypoint descriptors without gaze representation.
A.1. Reenactment performance
The section provides an additional qualitative comparison of talking-head systems in a cross-person reenactment scenario.
- Figure 6 presents an additional qualitative comparison of the listed talking-head systems for cross-person reenactment.
A.2. Ablation study
The ablation study examines pose dimensionality, encoder capacity, background handling, and pose augmentation through quantitative and qualitative comparisons. The experiments expose a trade-off between identity preservation and pose reconstruction, with the preferred configuration depending on the reenactment scenario.
- Ablation design: The study varies pose dimensionality, pose-encoder capacity, background handling, and pose augmentation, evaluating resulting models quantitatively and qualitatively.
- Pose vector dimensionality: Reducing the pose vector from 256 to 64 dimensions slightly increases pose reconstruction error while producing negligible qualitative pose differences.
- Pose encoder capacity: Increasing pose-encoder capacity improves pose accuracy but introduces person-specific features from the driver, substantially increasing identity error.
- Pose encoder capacity: A stronger pose encoder may therefore suit self-reenactment, where identity bleeding is less problematic.
- Background handling: Removing segmentation causes the pose encoder to spend capacity on background information instead of subtle facial-expression details.
- Pose augmentation: Removing pose augmentation improves pose representation but harms identity preservation and can make systems behave more like autoencoders.
- Overall trade-off: The experiments identify a trade-off between identity preservation error IT and pose reconstruction error PT, with different configurations favored for self-reenactment versus cross-person driving.