Source-linked AI summary
Face2Face: Real-time Face Capture and Reenactment of RGB Videos
Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner
TL;DR
Face2Face addresses real-time facial reenactment from monocular RGB video, transferring a live source actor’s expressions onto a target sequence. It combines non-rigid identity recovery, dense runtime tracking, deformation transfer, target-based mouth synthesis, and photo-realistic compositing. The authors demonstrate real-time Youtube reenactment and report improved quality and runtime over prior reenactment methods, while noting limitations from illumination assumptions, occlusions, model capacity, mouth-data sparsity, and hardware delay.
Problem
Real-time transfer of a live source actor’s expressions to arbitrary monocular target videos requires photo-realistic reenactment from RGB input rather than offline or depth-dependent processing.
Method
The system performs global non-rigid identity bundling, dense source-and-target expression tracking, low-dimensional deformation transfer, target-sequence mouth synthesis, and illumination-aware compositing.
Results
The authors demonstrate highly convincing real-time reenactment of Youtube videos from a webcam source and report better video quality and runtime than state-of-the-art reenactment methods.
Takeaways & Limitations
The approach provides a monocular RGB-only route for animating legacy video footage in real time, with applications including VR/AR, teleconferencing, and on-the-fly dubbing.
Takeaways & Limitations
The method assumes Lambertian surfaces and smooth illumination, uses a 76-coefficient low-dimensional model, requires visible target expression variation for mouth synthesis, and incurs approximately three frames of hardware delay.
Abstract
from arXiv · showhide
We present Face2Face, a novel approach for real-time facial reenactment of a monocular target video sequence (e.g., Youtube video). The source sequence is also a monocular video stream, captured live with a commodity webcam. Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion. To this end, we first address the under-constrained problem of facial identity recovery from monocular video by non-rigid model-based bundling. At run time, we track facial expressions of both source and target video using a dense photometric consistency measure. Reenactment is then achieved by fast and efficient deformation transfer between source and target. The mouth interior that best matches the re-targeted expression is retrieved from the target sequence and warped to produce an accurate fit. Finally, we convincingly re-render the synthesized target face on top of the corresponding video stream such that it seamlessly blends with the real-world illumination. We demonstrate our method in a live setup, where Youtube videos are reenacted in real time.
1. Introduction
Face2Face introduces real-time monocular RGB facial reenactment, transferring a live source actor’s expressions to arbitrary target videos and rendering the result photo-realistically. Its pipeline combines dense tracking, identity recovery, expression transfer, and target-sequence mouth synthesis.
- Contribution: Face2Face transfers a source actor’s facial expressions to an arbitrary monocular target video in real time.The source is captured live with a commodity webcam, while the target may be legacy footage such as a Youtube video.
- Contribution: The method uses dense markerless facial performance capture from monocular RGB data rather than transferring expressions to virtual CG characters.The paper positions monocular facial reenactment as its main contribution.
- Method: A global non-rigid model-based bundling step reconstructs target identity from a prerecorded training sequence and resolves monocular reconstruction ambiguities.Identity parameters are estimated globally over training frames before runtime tracking.
- Method: At runtime, dense analysis-by-synthesis tracking estimates source and target expressions, followed by efficient deformation transfer in a low-dimensional expression space.The synthesized target face is rendered with transferred expression coefficients and composited using estimated environment lighting.
- Results: The live setup demonstrates webcam-driven reenactment of Youtube videos, with the authors reporting higher video quality and faster runtime than state-of-the-art reenactment methods.The paper presents this as the first real-time RGB reenactment method.
- Method: The system synthesizes the target mouth interior from the target sequence instead of directly copying the source mouth or using a generic teeth proxy.This addresses shortcomings identified in prior online reenactment approaches.
3. Synthesis of Facial Imagery
The facial synthesis model separates identity, reflectance, and expression using a multilinear PCA representation, then renders faces under pose, illumination, and camera parameters.
- Model: The multilinear PCA model uses shape, albedo, and expression parameters to represent facial imagery.The first two dimensions encode facial identity through geometric shape and skin reflectance, while the third controls expression.
- Model: Mgeo(α, δ) = aid + Eid · α + Eexp · δ combines average identity shape with identity and expression basis deformations.α controls identity and δ controls facial expression in the geometric model.
- Model: The statistical prior models shape and reflectance around average values using identity, albedo, and expression bases with corresponding standard deviations.The face model contains 53K vertices and 106K faces.
- Rendering: Image synthesis depends on face parameters α, β, δ, illumination γ, rigid transformation R,t, and camera parameters κ.The unknown parameter vector P is the union of these quantities.
4. Energy Formulation
The energy formulation jointly fits synthesized imagery and facial features to RGB input while regularizing parameters toward plausible faces. A robust GPU-based optimization minimizes this nonlinear objective in real time.
- Objective: The objective jointly reconstructs unknown parameters from monocular input using photo-consistency, feature alignment, and statistical regularization.The formulation is a robust variational optimization problem.
- Objective: The data term compares synthesized imagery with input data through photo-consistency and facial feature alignment, with weights wcol = 1, wlan = 10, and wreg = 2.5 · 10^-5.These weights balance the color, landmark, and prior sub-objectives.
- Photo-Consistency: Ecol measures pixel-level photometric alignment between the synthesized and input RGB images over visible pixels.The method uses an ℓ2,1-norm rather than least squares to improve robustness to outliers.
- Feature Alignment: Feature alignment matches detected salient facial points to corresponding model vertices, weighted by detection confidence.This term helps avoid local minima in the highly nonlinear color-consistency objective.
- Objective: The regularization term keeps synthesized face parameters statistically close to the mean and helps prevent degenerate geometry and reflectance.It also guides optimization away from local minima.
5. Data-parallel Optimization Strategy
The method uses data-parallel optimization to solve robust tracking and non-rigid model-based bundling problems in real time. IRLS, Gauss–Newton, and PCG enable efficient updates over tracking parameters and keyframes.
- A GPU-based IRLS solver minimizes the robust tracking objective in real time by repeatedly converting it into nonlinear least-squares problems.Each iteration splits the norm into components, keeping one part fixed while updating it afterward.
- Each IRLS iteration performs one Gauss–Newton step and solves the normal equations with PCG for the linear parameter update.The Jacobian and right-hand side are precomputed and stored in device memory.
- Non-rigid model-based bundling jointly estimates identity, camera intrinsics, pose, and illumination across k keyframes.The approach addresses the under-constrained identity-estimation problem in monocular reconstruction.
7. Expression Transfer
Expression transfer operates directly in the facial model’s low-dimensional expression space. The resulting least-squares system supports precomputation and real-time computation of the target expression.
- Sub-space deformation transfer preserves person-specific expression characteristics while transferring source expression changes to the target actor.It operates directly in the expression blendshape space rather than full mesh space.
- The transferred target expression δT is produced directly in the reduced sub-space of the parametric facial prior.
- The target deformation is found by solving a linear least-squares problem based on source deformation gradients and the target’s undeformed state.The formulation uses triangle edge matrices for the source and target meshes.
- A constant system matrix enables precomputation of its pseudoinverse with SVD, leaving a small 76 × 76 system to solve in real time.No additional smoothness term is required because the blendshape model implicitly provides one.
- Mouth-frame retrieval uses an appearance graph that balances similarity to the previously retrieved frame against distance to the target expression.
8. Mouth Retrieval
The mouth-retrieval pipeline selects target-sequence frames that match the transferred expression while maintaining temporal coherence and target-mouth appearance. Clustering, appearance-graph matching, blending, and compositing support real-time reenactment.
- The method retrieves and warps the best matching mouth image from the target sequence while preserving the target actor’s mouth appearance.The authors assume that sufficient mouth variation is available in the target video.
- The retrieval descriptor combines rotation, expression parameters, landmarks, and local binary patterns to measure geometric and photometric similarity.
- Frame-to-cluster matching uses ten modified k-means clusters and selects representative frames to enable real-time retrieval.Each representative minimizes its distance to the other frames in its cluster.
- A fully connected appearance graph finds an intermediate frame that is similar to both the last retrieved frame and the current target frame.Its edge weights incorporate RGB cross-correlation, parameter-space distance, and landmark distance.
- Table 1 reports average run times for three sequences, with final-frame-rate standard deviations of 0.51, 0.56, and 0.59 fps.CPU and GPU stages run in parallel.
- The pipeline blends successive mouth frames after optical-flow alignment, applies illumination correction, and composites the corrected mouth with the original frame and rendered face.
9. Results
Face2Face achieves real-time reenactment on commodity hardware using monocular RGB source and target videos. Its tracking is competitive with RGB and RGB-D methods, while reenactment quality and runtime compare favorably with prior approaches.
- Live Reenactment Setup: The live setup captures a source stream with a commodity webcam and reenacts downloaded monocular YouTube target videos.Experiments use a Logitech HD Pro C920 at 30Hz and 640 × 480 resolution.
- Runtime: The method runs in real time on a commodity desktop computer with an NVIDIA Titan X and Intel Core i7-4770.The pipeline includes tracking and additional reenactment stages.
- Tracking Comparison to Previous Work: Our tracking is on par with or exceeds the state of the art across monocular RGB, RGB-D, and offline monocular comparisons.The comparisons cover Cao et al., Thies et al., FaceShift, and Shi et al.
- Tracking Comparison to Previous Work: Compared with Shi et al., the online approach yields a closer face fit, particularly at the silhouette of the input face.The authors attribute this to denser non-rigid bundle adjustment producing a better shape identity estimate than the sparse approach.
- Tracking Comparison to Previous Work: Compared with Cao et al., the methods usually produce similar high-quality results, while Face2Face estimates identity and expression slightly more accurately.Compared with Thies et al. and FaceShift, tracking accuracy is similar without requiring depth data.
- Reenactment Evaluation: Against prior reenactment methods, Face2Face provides highly realistic output in real time, whereas Garrido et al. is fundamentally offline and Thies et al. requires RGB-D input.The comparisons also report undesired mouth appearance changes from geometric teeth proxies used by the prior methods.
10. Limitations
The method has limitations from its illumination and surface assumptions, model capacity, sequence content, and commodity hardware latency.
- Lambertian surfaces and smooth illumination can produce artifacts under hard shadows or specular highlights.
- Long-hair and beard occlusions remain challenging for the system.
- The 76-coefficient blendshape model omits fine-scale static and transient surface details.
- Mouth synthesis requires sufficient visible expression variation; short or static target sequences can cause temporal aliasing.
- Commodity webcam, USB, and PCI hardware introduce a delay of approximately 3 frames.
11. Conclusion
Face2Face is presented as the first real-time facial reenactment system using only monocular RGB input. Its live setup animates legacy video footage in real time and is positioned for VR/AR, teleconferencing, and translated-audio dubbing.
- Face2Face is the first real-time facial reenactment system requiring only monocular RGB input.
- The live setup enables real-time animation of legacy video footage, including Youtube videos.
- The authors identify VR/AR, teleconferencing, and on-the-fly dubbing with translated audio as potential applications.
A. Optimization Framework
The optimization framework combines GPU-parallel Gauss-Newton optimization with promoted global and per-frame parameters, hierarchical coarse-to-fine processing, and PCG-based normal-equation solving.
- The GPU implementation parallelizes visible-pixel gathering, Jacobian computation, and gradient evaluation across processors.
- Normal equations are solved with preconditioned conjugate gradient using a Jacobi preconditioner and two matrix-vector products.
- The implementation uses DirectX and DirectCompute to reduce rendering-to-compute resource-mapping overhead.
- The framework exploits a block-dense Jacobian structure and promoter functions to assemble per-frame contributions in the global parameter space.
- Global identity and camera parameters are shared across frames, while expression, rigid pose, and illumination parameters are frame-local.
- Coarse-to-fine hierarchical optimization helps avoid local minima by propagating solutions from coarse to finer levels.
- Using k = 6 keyframes, identity estimation takes approximately 20 seconds, while processing time scales linearly with the number of keyframes.
C. Reenactment Evaluation
The evaluation compares Face2Face with prior reenactment systems and measures self-reenactment photometric error. Results emphasize target-identity preservation, competitive visual quality, and low reconstruction error.
- The online approach achieves similar or better quality than Garrido et al.’s offline self-reenactment method.
- Compared with Dale et al. and Garrido et al., Face2Face preserves the target identity while altering expression according to the source actor.
- Self-reenactment cross-validation yields a mean photometric error of 0.33 pixels in the first half and 0.42 pixels in the second half.The corresponding standard deviations are 0.157px and 0.17px, respectively.
- Table 2 lists the referenced Youtube videos used in the evaluation.
- The face model represents shape, albedo, and expression using parametric bases and associated standard deviations.
- Deformation transfer is formulated through an energy with a system matrix and right-hand side.