Source-linked AI summary
Deep Video Portraits
Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Nießner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, Christian Theobalt
TL;DR
The paper addresses the limited control of existing portrait-video reenactment methods, which mainly manipulate facial expressions. It introduces a space-time rendering-to-video network that converts synthetic parametric-face renderings into photorealistic target frames, enabling transfer of full head motion, expressions, gaze, and blinking. Experiments and a user study demonstrate high-fidelity reenactment and video-editing applications, while quality degrades for poses outside the training corpus and output resolution remains medium.
Problem
Existing approaches are largely restricted to facial-expression manipulation, whereas the paper targets transfer of full head pose, expression, eye gaze, and blinking between portrait videos.
Method
A space-time conditional generative network translates synthetic renderings of a parametric target face and eye gaze into temporally coherent, photorealistic full-frame video.
Results
The method transfers full 3D head pose, facial expression, eye gaze, and blinking while synthesizing consistent upper body, hair, and background, with user studies confirming high fidelity.
Takeaways & Limitations
Freely recombining source and target parameters enables full-head reenactment, visual dubbing, and interactive portrait-video editing without explicitly modeling hair, body, or background.
Takeaways & Limitations
Quality can degrade for extreme head poses or expressions outside the training corpus, and the method cannot actively control torso, hair, or background motion.
Abstract
from arXiv · showhide
We present a novel approach that enables photo-realistic re-animation of portrait videos using only an input video. In contrast to existing approaches that are restricted to manipulations of facial expressions only, we are the first to transfer the full 3D head position, head rotation, face expression, eye gaze, and eye blinking from a source actor to a portrait video of a target actor. The core of our approach is a generative neural network with a novel space-time architecture. The network takes as input synthetic renderings of a parametric face model, based on which it predicts photo-realistic video frames for a given target actor. The realism in this rendering-to-video transfer is achieved by careful adversarial training, and as a result, we can create modified target videos that mimic the behavior of the synthetically-created input. In order to enable source-to-target video re-animation, we render a synthetic target video with the reconstructed head animation parameters from a source video, and feed it into the trained network -- thus taking full control of the target. With the ability to freely recombine source and target parameters, we are able to demonstrate a large variety of video rewrite applications without explicitly modeling hair, body or background. For instance, we can reenact the full head using interactive user-controlled editing, and realize high-fidelity visual dubbing. To demonstrate the high quality of our output, we conduct an extensive series of experiments and evaluations, where for instance a user study shows that our video edits are hard to detect.
1 INTRODUCTION
The paper addresses photo-realistic video portrait synthesis and reenactment, transferring controllable head motion and eye behavior from a source actor to a target. Its rendering-to-video approach generates complete target portraits, including upper body, hair, clothing, and consistent backgrounds.
- Photo-realistic portrait reenactment transfers a source actor’s actions to a target actor, addressing applications such as video editing, visual effects, dubbing, virtual reality, and telepresence.
- The generated frames support joint or independent manipulation of head pose, facial expression, eye motion, and, to some extent, face identity.
- Existing face-interior reenactment methods realistically modify facial expressions but generally do not provide full 3D head pose with consistent upper body and changing background.
- The method performs rendering-to-video translation from coarse, controllable target face-model renderings and separately rendered eye-gaze images to complete photorealistic portraits.
- The target and scene-specific networks require only a few minutes of portrait footage for training and synthesize realistic clothing, hair, upper body, and scene background under full 3D head control.
- A novel space-time encoding represents face geometry, reflectance, motion, eye gaze, and eye blinks as conditional input for temporally coherent video synthesis.
2 RELATED WORK
Prior work largely targets facial-expression manipulation, stylized head animation, or generic image translation. Deep Video Portraits instead generates temporally coherent, photorealistic portraits that follow a source actor’s head pose, expression, and eye gaze.
- The work combines face reconstruction, facial reenactment, visual dubbing, and conditional image-to-image translation into a full-portrait video synthesis task.
- Unlike prior methods, it provides a generative model covering the entire head, hair, mouth interior, and eye gaze.
- The proposed approach synthesizes temporally coherent video portraits that follow a source actor’s head pose, facial expression, and eye gaze.
3 OVERVIEW
The system tracks source and target videos with a parametric face and illumination model, then transfers selected source parameters into target conditioning inputs. A generator synthesizes full target frames while preserving target identity and appearance.
- Monocular face reconstruction produces low-dimensional parameter sequences for the source and target actors.
- The system transfers rigid head pose, facial expression, and eye motion while preserving the target actor’s identity and appearance.
- Full target frames are synthesized with consistent upper-body posture, hair, and background.
4 MONOCULAR FACE RECONSTRUCTION
The method reconstructs source and target facial performances with a parametric face model, estimating pose, identity, expression, gaze, and illumination from monocular video. It combines analysis-by-synthesis fitting, photometric and landmark alignment, and statistical regularization.
- Dense Face Reconstruction: The reconstructed parameter sequence encodes head rotation and translation, identity geometry and reflectance, facial expression, binocular gaze, and spherical-harmonics illumination.The representation uses 80 geometry coefficients, 80 reflectance coefficients, 64 expression coefficients, 4 gaze parameters, and 27 illumination coefficients.
- Parametric Face Representation: The parametric representation models facial geometry, skin reflectance, and expression with separate low-dimensional bases.Geometry and reflectance use affine models, while expression variation is represented through a separate basis.
- Image Formation Model: Rendering assumes a fixed identical camera across scenes and models scene motion through the face, with distant illumination approximated using spherical harmonics.For diffuse materials, three spherical-harmonics bands provide an average approximation error below 1 percent and yield 9 parameters per color channel.
- Dense Face Reconstruction: Face reconstruction fits the model to each frame by maximizing photometric consistency while combining dense photo-consistency, landmark alignment, and statistical regularization.The procedure uses 66 automatically detected landmarks, robust ℓ1 photometric alignment, and a regularizer enforcing statistically plausible parameters.
- Dense Face Reconstruction: Identity is estimated only in the first frame and then held constant, while other facial parameters are estimated for every frame.The optimization uses a data-parallel implementation of iteratively re-weighted least squares.
5 SYNTHETIC CONDITIONING INPUT
The system creates controllable synthetic conditioning sequences for a target actor by transferring source animation parameters relative to a neutral reference. Each temporal window combines color, correspondence, and eye-gaze renderings before translation into video.
- Parameter Transfer: For target reenactment, source pose, expression, and gaze parameters are copied relative to a neutral reference and applied to the target sequence.The target identity remains fixed while the selected animation parameters are modified.
- Conditioning Images: Each conditioning frame contains a color rendering under target illumination, a correspondence image, and an eye-gaze image.These inputs respectively depict the modified target model, projected face-model vertex identities, and pupil locations within the eye regions.
- Conditioning Images: The color rendering preserves target identity and provides a close starting point because only the face-region difference from a real image must be learned.The correspondence image uses a constant unique gradient texture to encode projected parametric vertex indices.
- Space-Time Conditioning: A sliding-window space-time tensor stacks 3-channel conditioning images across Nw frames, producing an input with dimensions W × H × 9Nw.The generated output frames are assembled directly to form the complete photorealistic video.
6 RENDERING-TO-VIDEO TRANSLATION
A conditional rendering-to-video network converts synthetic space-time face inputs into photorealistic target frames that reproduce the specified motion and gaze. Its adversarial, reconstruction-based training combines temporal conditioning with an encoder-decoder architecture and extended PatchGAN discrimination.
- Rendering-to-Video Translation: The method synthesizes the entire foreground actor, including face, hair, and other actor regions, from synthetic conditioning inputs.The generated target actor mimics the head motion, facial expression, and eye gaze represented by the synthetic input.
- Network Architecture: The transformation network maps a W × H × 9Nw conditioning tensor to a photorealistic target image, using temporal history to synthesize coherent video frames.The temporal axis is aligned with network channels, allowing the first convolution to process 9Nw channels.
- Network Architecture: The network uses an encoder-decoder with skip connections and cascaded refinement to preserve fine-scale structure and produce high-resolution frames.The decoder upsamples and applies convolutional refinement before mapping outputs into normalized [−1, +1]-space.
- Adversarial Training: The discriminator receives the conditioning volume together with either the generated or ground-truth image, extending PatchGAN-style classification to conditioning-image volumes.This setup evaluates whether the output is real or synthetic in the context of its conditioning input.
- Objective Function: Training minimizes an adversarial loss together with an ℓ1 reproduction loss weighted by λ=100.The adversarial term promotes realism, while the ℓ1 term penalizes distance from the ground-truth image and encourages sharpness.
- Training: Approximately 2,000 video frames, or about one minute of footage, are typically sufficient to train a target-specific network with an 11-frame window.Training uses rendered conditioning volumes paired with corresponding ground-truth images.
7 RESULTS
The results show that the approach supports controllable, photorealistic portrait rewriting across full-head reenactment, facial reenactment, visual dubbing, interactive editing, and comparative evaluations. Outputs are temporally coherent, plausible beyond the conditioned face region, and difficult for users to distinguish from real videos.
- Comparisons: Against nearest-neighbor retrieval in pose-expression parameter space, the method provides higher-quality, more temporally coherent results without abrupt neighbor changes.Nearest-neighbor outputs can mismatch pose or expression and become unstable, especially with small training sets.
- Full-head reenactment: Full-head reenactment transfers a source actor’s 3D head pose, facial expression, eye gaze, and blinking to a target while retaining target mannerisms.The synthesized head and scene respond consistently, including background-shadow motion.
- Full-head reenactment: The approach produces nearly indistinguishable self-reenactment results when trained on two thirds of a target video and tested on the remaining third.The evaluation uses rendered conditioning images from tracked training and driving sequences.
- Facial reenactment: Facial reenactment transfers source expressions while preserving the target’s head pose, position, and eye gaze.The target expression coefficients are replaced before synthesizing the conditioning input.
- Visual dubbing: Visual dubbing achieves high-quality mouth-motion transfer while retaining the target’s gaze and eye blinks, with realistic mouth interiors and emotional content.The target’s original eye-gaze images remain part of the conditioning input.
- Interactive editing: Interactive editing changes expression, head rotation, translation, eye gaze, and blinks independently or together while generating consistent torso, hair, background, shadows, and reflections.The editor runs at approximately 9 fps.
- Quantitative evaluation: Space-time conditioning lowers photometric error and improves temporal stability, while errors remain higher in regions not explained by the conditioning input.Synthesized hair and upper body remain plausible and consistent despite differences outside the face region.
- User study: 47% of participant ratings judged the synthesized clips real across scenarios, compared with 80% for real clips.The study included 1,767 synthesized-video ratings and 1,362 real-video ratings.
8 DISCUSSION
The approach produces highly realistic reenactment within the training corpus, but its quality degrades for extreme poses or expressions and its controllability is limited beyond tracked facial motion.
- Extreme target head poses or facial expressions outside the training corpus can degrade the visual quality of generated portraits.Artifacts are most prominent outside the face region because those regions receive no conditioning input.
- The method does not actively control torso motion, hair motion, or the background because it tracks only the face with a parametric model.The network instead extrapolates a plausible upper body and background for a given head pose.
- The approach currently produces only medium-resolution output because of memory and training-time limitations.This makes fine-scale details such as individual teeth difficult to reproduce temporally coherently.
- Body tracking and an underlying body model could extend the conditioning inputs and address some control limitations.
- The authors note that advanced video editing capabilities increase the need for verifiable video authenticity, such as invisible watermarking.
9 CONCLUSION
The paper presents a rendering-to-video network that transfers head pose, facial expression, and eye gaze between actors while generating photorealistic, temporally coherent portraits. Experiments and a user study report higher quality and broader capabilities than prior work, supporting applications including reenactment, telepresence, editing, and dubbing.
- The method transfers head pose and orientation, facial expression, and eye gaze from a source actor to a target actor.
- A rendering-to-video network converts simple computer-graphics renderings into photorealistic, temporally coherent video using space-time conditioning.
- Experiments and a user study report that the method outperforms prior work in quality and expands the available reenactment possibilities.
- The approach supports applications including virtual-reality and telepresence reenactment, interactive video editing, and visual dubbing.
A APPENDIX
The appendix documents the datasets used for evaluation, listing target and source videos with their sequence lengths and associated footage credits.
- The appendix describes the datasets used and refers to separate tables for target actors and source actors.
- Table 4 lists target video names and sequence lengths in frames, together with footage credits.
- The listed video names include Putin, Elizabeth II, Reagan, Thomas, and Wolf.
- Table 5 lists source video names and sequence lengths in frames, with footage credits.