Source-linked AI summary
Automatic Face Reenactment
Pablo Garrido, Levi Valgaerts, Ole Rehmsen, Thorsten Thormaehlen, Patrick Perez, Christian Theobalt
TL;DR
Video face reenactment must preserve a target actor’s performance while replacing the actor’s face, despite limited source expressions and differing recordings. The paper proposes a fully automatic image-based pipeline combining retrieval and 2D face transfer, and reports convincing results on recorded and low-quality footage. Its main scope boundary is sensitivity to local lighting changes and mouth-region artifacts.
Problem
Video reenactment requires preserving target facial performance while replacing the face, yet existing techniques often demand manual frame-by-frame interaction.
Method
The system retrieves source frames using temporal clustering and a combined appearance-and-motion metric, then transfers the face through identity-preserving 2D warping.
Results
The authors report convincing reenactment results from short source videos with arbitrary facial expressions, including existing and low-quality footage.
Takeaways & Limitations
Fully automatic image-based reenactment can preserve target performance without a source-expression database or complex 3D face model.
Takeaways & Limitations
Local lighting variations can cause incorrect seam color propagation, flicker, less realistic reenactments, and mouth ghosting; the authors plan separate mouth driving and more robust compositing.
Abstract
from arXiv · showhide
We propose an image-based, facial reenactment system that replaces the face of an actor in an existing target video with the face of a user from a source video, while preserving the original target performance. Our system is fully automatic and does not require a database of source expressions. Instead, it is able to produce convincing reenactment results from a short source video captured with an off-the-shelf camera, such as a webcam, where the user performs arbitrary facial gestures. Our reenactment pipeline is conceived as part image retrieval and part face transfer: The image retrieval is based on temporal clustering of target frames and a novel image matching metric that combines appearance and motion to select candidate frames from the source video, while the face transfer uses a 2D warping strategy that preserves the user's identity. Our system excels in simplicity as it does not rely on a 3D face model, it is robust under head motion and does not require the source and target performance to be similar. We show convincing reenactment results for videos that we recorded ourselves and for low-quality footage taken from the Internet.
1. Introduction
The paper presents a fully automatic, image-based video face-reenactment system that inserts a user's face while preserving the target actor's facial performance. It combines appearance-and-motion retrieval, temporal clustering, and 2D face transfer without requiring similar performances or a 3D face model.
- Video face reenactment replaces a target actor’s face while preserving the actor’s gestures and facial expressions.Existing techniques often require careful frame-by-frame analysis and manual interaction to generate smooth composites.
- The proposed method automatically inserts a user’s face into an existing target sequence while conserving the target’s performance, scene appearance, and lighting.It replaces only the inner face region and preserves the hair, face outline, skin color, background, and illumination.
- The pipeline accepts a short source video of arbitrary facial gestures and does not require source and target performances to have similar pose, timing, or expression.It is designed for automatic reenactment from ordinary footage, including webcam recordings.
- The system uses a simple image-based warping strategy that preserves facial identity without relying on a 3D face model.This avoids constructing and tracking a personalized face model.
- A novel matching metric combines facial appearance and motion, while temporal clustering stabilizes source-frame selection.The approach retrieves similar expressions while accounting for temporal continuity.
2. Related Work
Related work spans facial puppetry, face swapping, 3D face-model methods, and image-based approaches. The paper positions its method as a fully automatic, purely image-based alternative for reenactment under differing poses and performances.
- Facial puppetry transfers a user’s expressions to a virtual character, whereas face swapping exchanges faces between images or videos.These categories support applications including avatar animation and online identity protection.
- 3D morphable-model methods parameterize identity and expression, enabling accurate meshes and one-to-one expression mappings.Generating a detailed personalized model can be time consuming and may require extensive example data.
- Image-based methods match faces using image information, often combining 2D tracking with expression retrieval from a database.Many such systems assume similar frontal poses and allow limited head motion.
- The paper identifies Dale et al.’s 3D morphable-model technique as the only prior approach known to support face reenactment.That approach is described as complex, not fully automatic, pose-constrained, and mainly demonstrated on similar performances.
- The proposed method is presented as purely image-based, fully automatic, less complex, and robust to head-pose changes and differing performances.Its retrieval step is designed for source and target sequences with notably different performances.
3. Overview of our Face Reenactment System
The system takes source and target performance videos and produces a reenacted sequence that preserves the target performance while inserting the source user’s inner face. Its pipeline consists of tracking, matching, and transfer.
- The inputs are a source sequence of a user and a target sequence of an actor; the output replaces the actor’s inner face while preserving target performance and scene appearance.The source and target need not depict the same performance.
- A single short source video of random facial expressions can support reenactments for different target videos.Target footage may contain varied expressions and head poses.
- Face Tracking: Face tracking estimates and stabilizes facial landmarks for the user and actor throughout both videos.The tracking stage provides annotated frames for subsequent processing.
- Face Matching: Face matching retrieves source frames using local-binary-pattern appearance features, temporal target chunks, and face motion.Chunking and motion information make matching more stable and temporally coherent.
- Face Transfer: Face transfer warps selected source landmarks to the target pose, synthesizes in-between frames, and blends the source face into the target sequence.The transfer stage creates smooth transitions using seamless cloning.
4. Non-Rigid Face Tracking
The tracking stage estimates non-rigid 2D facial landmarks for both videos using a constrained local model, then corrects noisy locations to improve downstream stability.
- The tracker provides 66 consistent landmark locations covering facial features such as the eyes, nose, mouth, and face outline.These landmarks are used to represent facial shape.
- The tracking method is a constrained local model using subspace constrained mean-shift optimization and a 3D point distribution model.The model represents non-rigid shape variation around 3D reference landmarks.
- The tracker composes non-rigid shape variations with global scale, rotation, and translation parameters to estimate 2D landmark locations.The paper uses only the resulting 2D landmark output downstream.
- Optical-flow correction refines landmarks between reliable automatically selected key frames.This addresses inaccuracies for expressions on which the tracker was not trained and improves matching and transfer stability.
5. Face Matching
The system matches source and target faces using appearance and landmark motion while clustering target frames temporally to improve stability. It aligns facial regions, compares target clusters with source frames, and selects matches that accommodate different timing and speed.
- Matching metric: Face matching combines aligned facial appearance with landmark motion to balance expression similarity against temporal stability under head-pose differences.Appearance describes facial expression, while landmark motion captures expression change and its rate.
- Feature extraction: The appearance descriptor uses LBP histograms extracted from tiled mouth, eye, and nose regions after 2D alignment to a common reference.Figure 2 shows the landmark alignment, triangulation, and region tiling used for feature extraction.
- Temporal clustering: Target frames are grouped into temporally continuous clusters of similar expression and appearance using hierarchical agglomerative clustering.Clusters merge consecutive frames while preserving temporal continuity and limiting within-cluster variation.
- Frame selection: For each target cluster, the system aggregates appearance and motion distances across frames and selects the source frame with the smallest total distance.The selected source frame is assigned to the cluster's central timestamp.
- Matching metric: Because source frames need not be consecutive, the matching metric supports source and target sequences with different timing and speed.Appearance and motion distances are normalized to [0, 1], with τ set to 0.8 in the experiments.
6. Face Transfer
The face-transfer stage estimates a user-face shape matching each target pose while preserving identity, then transfers appearance through landmark-based warping, interpolation, and seamless blending.
- Shape Transfer: The method combines non-rigid and affine terms to deform the user’s facial landmarks toward the target shape while retaining source identity.The non-rigid term penalizes target-shape deviations, while the affine term penalizes deviations from selected source shapes.
- Shape Transfer: Selected source shapes immediately before and after each target time are affine-aligned and interpolated to preserve the user’s identity.Their weights depend linearly on the distance to the temporal-cluster centers.
- Appearance Transfer: The optimal reenactment shape is used to inverse-warp source textures through a triangulation of landmark points.For in-between frames, appearance is interpolated from the closest selected source frames using the same landmark triangulation.
- Seam Generation: The system generates a tight source mask, transfers it by inverse-warping, intersects the seam with a target-face mask, and blends the result using Poisson cloning.Boundary pixels are additionally blended with a Gaussian whose standard deviation is 9 pixels.
- Shape and Appearance Transfer: The proposed warping avoids distortions from global non-rigid warping and odd deformations from global affine warping under perspective or mismatched shapes.Global non-rigid warping fits the source shape exactly to the target, while global affine warping can fail under strong perspective views.
7. Results
The system produces convincing reenactments on studio footage and low-quality Internet videos, including cases with substantial differences in speech, timing, lighting, resolution, and head motion. A user study supports temporal clustering and combined appearance-motion matching, while remaining limitations include lighting sensitivity, mouth ghosting, and expression mismatches.
- Low-Quality Internet Video: The system produced plausible animations for 18 s and 8 s Internet-video subsequences from a 10 s webcam recording despite differing speech, head pose, lighting, and resolution.Face matching and transfer took between 4 and 7 min.
- Validation: 3.25 average score for the full system exceeded 2.92 without temporal clustering and 1.48 without motion distance across seven videos rated by 32 participants.Ratings covered mimicking fidelity, temporal consistency, and visual artifacts on a 1-to-5 scale.
- Additional Analysis: Increasing source-frame quantity improved reenactment realism and temporal smoothness; same-sequence self-reenactments produced 1 and 36 mismatches across 59 and 214 clusters.The supplementary analysis reports that mismatches were mostly visually similar to their cluster centers.
- Comparison: The method was visually close in quality to Dale et al.’s semi-automatic 3D result, although frame-level outputs differed because the systems preserve different performances.The comparison used data depicting two subjects reciting the same poem.
- Limitations: Credible animations require constant or globally changing lighting; local variations can cause seam color errors, flicker, less realistic results, and mouth ghosting.The discussion also identifies blending and temporal inconsistencies as sources of artifacts.
8. Conclusion
The paper presents a fully automatic, image-based reenactment system that preserves target facial performance while replacing the inner face. Its experiments show convincing results from short videos of arbitrary facial expressions without user interaction or a complex 3D model.
- Conclusion: The system replaces an actor’s inner face while preserving the original facial performance without user interaction or a complex 3D face model.It combines expression matching, temporal clustering, appearance-motion matching, and image warping to handle moderate head motion.