Source-linked AI summary
On Face Segmentation, Face Swapping, and Face Perception
Yuval Nirkin, Iacopo Masi, Anh Tuan Tran, Tal Hassner, Gerard Medioni
TL;DR
The paper addresses face swapping between unconstrained, arbitrarily paired images, where differing appearances and contexts make robust processing difficult. It uses a standard FCN trained on richly generated examples within a robust swapping pipeline, then evaluates intra- and inter-subject effects on LFW recognition. Intra-subject swaps remain as recognizable as their sources, while better inter-subject swaps produce less recognizable results.
Problem
Unconstrained, arbitrarily paired face images lack matched viewpoints, expressions, shapes, genders, and other attributes, motivating robust face swapping and quantitative recognition tests.
Method
The paper trains a standard FCN on challenging segmentation examples generated with motion cues and 3D augmentation, then uses the segmentations in a robust face-swapping pipeline.
Results
Intra-subject swaps have little effect on face verification accuracy, whereas inter-subject swapping changes faces and makes them less recognizable.
Takeaways & Limitations
Better face swapping produces more facial changes and a drop in recognition, providing quantitative support for the reported perceptual effect in machine vision systems.
Takeaways & Limitations
Inter-subject tests require matching swapped images carefully because comparing swapped-image pairs can obscure changes that make both source faces equally unrecognizable.
Abstract
from arXiv · showhide
We show that even when face images are unconstrained and arbitrarily paired, face swapping between them is actually quite simple. To this end, we make the following contributions. (a) Instead of tailoring systems for face segmentation, as others previously proposed, we show that a standard fully convolutional network (FCN) can achieve remarkably fast and accurate segmentations, provided that it is trained on a rich enough example set. For this purpose, we describe novel data collection and generation routines which provide challenging segmented face examples. (b) We use our segmentations to enable robust face swapping under unprecedented conditions. (c) Unlike previous work, our swapping is robust enough to allow for extensive quantitative tests. To this end, we use the Labeled Faces in the Wild (LFW) benchmark and measure the effect of intra- and inter-subject face swapping on recognition. We show that our intra-subject swapped faces remain as recognizable as their sources, testifying to the effectiveness of our method. In line with well known perceptual studies, we show that better face swapping produces less recognizable inter-subject results. This is the first time this effect was quantitatively demonstrated for machine vision systems.
1. Introduction
The paper presents a face-swapping method for unconstrained, arbitrarily paired images, combining standard-FCN segmentation with robust pipeline components and quantitative LFW evaluation. It finds that intra-subject swaps preserve recognition, whereas inter-subject swaps reduce recognizability as faces change to blend naturally with new contexts.
- Face swapping is useful for privacy preservation, digital forensics, and face-specific data augmentation when training data is scarce.
- Arbitrary unconstrained pairings remove guarantees of similar viewpoints, expressions, 3D shapes, genders, or other attributes, demanding robust face-processing methods.
- A rich labeled dataset generated through motion cues and 3D augmentation trains a standard FCN that surpasses previous segmentation results in accuracy and speed.
- The method combines face segmentation and a swapping pipeline designed to produce high-quality results under challenging unconstrained conditions.
- Quantitative LFW protocols measure how intra- and inter-subject swapping affects face verification, addressing a gap in prior face-swapping evaluation.
- Intra-subject swapping has little effect on verification accuracy, while inter-subject swapping changes faces and makes them less recognizable.
2. Related Work
Earlier face-swapping and segmentation methods commonly constrained target selection, estimated facial structure, or used specialized and potentially impractical networks. The paper instead uses a standard FCN with novel training-data collection and augmentation for face segmentation.
- Earlier segmentation methods targeted facial regions, used example-based approaches, or alternated segmentation with landmark localization.
- Deep segmentation methods handled multiple facial regions but could be slow, while another approach achieved real-time speed with a deconvolutional network.
- This paper uses a standard FCN with novel training-data collection and augmentation to obtain challenging segmentation examples.
- Previous face-swapping systems often selected target images sharing source attributes such as facial tone, pose, or expression.
- Many prior methods estimated 3D facial shapes or dense 2D appearance models to map textures across different facial structures.
- A deep style-transfer approach required training a separate network for each source image, limiting practicality.
3. Swapping faces in unconstrained images
The method swaps faces from unconstrained, arbitrarily paired images by combining landmark-based 3D alignment, FCN face visibility segmentation, proxy-based warping, and blending. Rich semi-supervised and synthetic-occlusion training data support segmentation under challenging conditions.
- 3.1. Fitting 3D face shapes: Facial landmarks establish 3D pose and expression for a generic or regressed 3D face shape.The method uses landmarks to estimate pose and modify shape to account for expression.
- 3.3. Face swapping and blending: Aligned 3D face shapes act as proxies for efficiently warping the visible source face onto the target, after which the result is blended with target context.The pipeline masks rendered intensities using target segmentation before applying off-the-shelf blending.
- 3.1. Fitting 3D face shapes: The face shape combines a mean Basel face with shape and expression components, using subject-specific shape and expression parameters.The representation uses the Basel Face Model for shape and the 3DDFA Morphable Model for expressions.
- 3.2. Deep face segmentation: A standard FCN predicts per-pixel visible-face regions, provided it is trained on plentiful, rich, and varied examples.The system uses FCN-8s-VGG and emphasizes challenging training examples rather than a task-specific architecture.
4. Experiments
Experiments evaluate segmentation quality, unconstrained face-swapping realism, failure modes, and recognition effects using COFW and LFW. The results support fast segmentation, challenging qualitative swaps, and distinct intra- versus inter-subject recognition patterns.
- Experimental setup: The method is evaluated qualitatively and quantitatively on unconstrained images, with GPU swapping at 1.3 fps and CPU swapping at 0.8 fps.Experiments use MatConvNet, DLIB, OpenCV, and an Intel Core i7 4820K system with an NVIDIA GeForce Titan X.
- Face segmentation results: On COFW, the method is the fastest while achieving comparable segmentation results to the state of the art across IOU, global accuracy, and ave(face) recall.The evaluation uses 507 face photos and reports runtimes alongside the three segmentation metrics.
- Qualitative face-swapping results: Qualitative LFW swaps use randomly selected targets with widely different poses, occlusions, and expressions, extending demonstrations beyond prior reported settings.Figure 5 also samples variation in gender, ethnicity, age, and expression.
- Qualitative face-swapping results: The qualitative comparison with uses the same source-target pairs, with differences attributed to segmentation quality and visible in the face hairlines.Figure 7 identifies landmark localization errors as the most common failure cause, followed by resolution differences and difficult facial-hue blending.
- Quantitative tests: Inter-subject tests show that stronger source-face modification lowers recognizability, whereas context-preserving blending can improve recognition by retaining more contextual information.Face-preserving accuracy is about 8% higher with a generic shape and no segmentation, while estimating 3D shape alone lowers accuracy by about 3.5%.
- Quantitative tests: Intra-subject swapping causes only a 1% accuracy drop with similar nAUC relative to unchanged LFW images, indicating that source identities remain recognizable.Estimated and generic 3D shapes produce nearly identical accuracy in this setting.
5. Conclusions
The paper concludes that rich training examples let a standard FCN achieve state-of-the-art segmentation speed and accuracy, while robust swapping enables quantitative recognition tests. Better swapping changes faces more and reduces recognition, and 3D shape estimation further reduces source-face recognizability.
- 5. Conclusions: A standard segmentation network can surpass tailored methods in speed and accuracy when trained on rich, diverse examples.The authors also report that collecting such examples is easy.
- 5. Conclusions: Face swapping supports quantitative tests showing that both faces and their contexts contribute to recognition.The tests provide quantitative support for a longstanding claim by Sinha and Poggio.
- 5. Conclusions: Better swapping produces more facial changes and a drop in recognition, including when masking facial spoofing attacks.This conclusion is framed as relevant to biometric-system spoofing.
- 5. Conclusions: 3D face shape estimation better blends the two faces and produces less recognizable source faces.
A. Additional intra-subject qualitative results
Additional qualitative results illustrate intra-subject face swapping outcomes.
- A. Additional intra-subject qualitative results: Qualitative intra-subject face swapping results are presented.
B. Additional segmentation results
Additional qualitative results illustrate face segmentation on COFW and LFW data.
- B. Additional segmentation results: Qualitative segmentation results from the COFW data set are presented.
- B. Additional segmentation results: Qualitative segmentation results from the LFW data set are presented.
C. Qualitative ablation results
Qualitative ablation studies present inter-subject and intra-subject face swapping results.
- C. Qualitative ablation results: A qualitative ablation study presents inter-subject face swapping results.
- C. Qualitative ablation results: A qualitative ablation study presents intra-subject face swapping results.