Source-linked AI summary
Fake It Till You Make It: Face analysis in the wild using synthetic data alone
Erroll Wood, Tadas Baltrušaitis, Charlie Hewitt, Sebastian Dziadzio, Matthew Johnson, Virginia Estellers, Thomas J. Cashman, Jamie Shotton
TL;DR
Face-related computer vision has struggled to use synthetic data alone because the real–synthetic domain gap makes generalization difficult. The paper combines a procedural parametric 3D face model with high-quality artist-created assets to generate realistic, diverse, richly labelled training data. Models trained without real images achieve results comparable to the state of the art on face parsing and landmark localization, while the approach supports broader synthetic-data applications.
Problem
The domain gap between real and synthetic data makes generalization difficult, so face-related machine learning rarely uses synthetic data alone.
Method
The paper procedurally combines a parametric 3D face model with high-quality artist-created assets to render realistic and diverse synthetic face images.
Results
Synthetic-only training achieves results comparable with the state of the art for face parsing and landmark localization without using a single real training image.
Takeaways & Limitations
Synthetic data can replace real data for the evaluated face-analysis tasks and can support other face-related tasks requiring rich labels.
Takeaways & Limitations
The model cannot simulate low-neckline clothing, lacks expression-dependent wrinkling, and sometimes produces unusual combinations of independently sampled attributes.
Abstract
from arXiv · showhide
We demonstrate that it is possible to perform face-related computer vision in the wild using synthetic data alone. The community has long enjoyed the benefits of synthesizing training data with graphics, but the domain gap between real and synthetic data has remained a problem, especially for human faces. Researchers have tried to bridge this gap with data mixing, domain adaptation, and domain-adversarial training, but we show that it is possible to synthesize data with minimal domain gap, so that models trained on synthetic data generalize to real in-the-wild datasets. We describe how to combine a procedurally-generated parametric 3D face model with a comprehensive library of hand-crafted assets to render training images with unprecedented realism and diversity. We train machine learning systems for face-related tasks such as landmark localization and face parsing, showing that synthetic data can both match real data in accuracy as well as open up new approaches where manual labelling would be impossible.
1. Introduction
The paper argues that realistic, diverse synthetic face data can overcome the domain gap and support face analysis in the wild without real training images. It combines procedural 3D face generation with artist-created assets to produce richly labelled data for multiple tasks.
- Motivation: Synthetic data provides perfect labels, rich annotations, and controlled variation without the cost, slowness, and bias of collecting and labelling real images.These advantages are especially relevant to human-related computer vision, where fairness and deployment ethics are important.
- Approach: The method renders realistic, diverse training images by procedurally generating faces and combining them with varied identities, expressions, textures, hair, clothing, and environments.Figure 2 describes the randomization process used to construct expressive synthetic faces.
- Central claim: Once a sufficiently realistic synthetic framework is available, training data can address real-world problems in the wild without using real data.The framework requires substantial expertise and investment initially but enables broad data generation with minimal incremental effort.
- Results: Models trained on one generic synthetic dataset perform as accurately as models trained on task-specific real datasets, with results in line with the state of the art.The paper evaluates face parsing and landmark localization and argues that synthetic data can support additional face-related tasks.
- Contributions: The paper contributes a synthetic-data generation method, photorealism ablations, and a publicly available synthetic dataset.The dataset is provided through the project webpage.
2. Related work
Prior face-related work rarely uses synthetic data alone because realistic full-face generation is difficult and the domain gap hinders generalization. Existing alternatives address only parts of the face, lack diversity or rich labels, make minor image adjustments, or alter semantics during adaptation; this paper instead minimizes the gap at the source.
- Motivation: Diverse face datasets are difficult to collect and annotate because web crawling raises privacy and copyright concerns, while manual labels can be inconsistent.These challenges motivate augmenting or replacing real data with synthetic data.
- Prior work: Full-face synthesis has received relatively little attention because modeling the human head is complex.Synthetic data has been more common for object, scene, eye, hand, and full-body analysis tasks.
- 3D models: 3D morphable models provide consistent labels, but prior efforts often render only facial parts, limiting their use for whole-face tasks.Examples include eye-region and hockey-mask models, as well as models for detailed facial geometry and gaze estimation.
- 3D scans: Directly rendering 3D scans can be realistic, but scan diversity limits variation and the scans cannot provide rich semantic machine-learning labels.Prior scan-based work includes dense 3D face alignment and head pose estimation.
- 2D methods: 2D image manipulation methods make only minor adjustments to existing images, limiting their usefulness for broader face-analysis training.Examples include head-pose warping and compositing hands onto faces for detection.
- Domain gap: Face-related synthetic data is rarely used alone because the domain gap between real and synthetic distributions makes generalization difficult.Instead, prior systems adapt synthetic data, mix it with real data, or use it for pre-training or regularization.
- Domain adaptation: Domain adaptation can change image semantics, making these techniques unsuitable for fine-grained annotations such as per-pixel labels and precise landmark coordinates.Regularization is used to preserve annotations or identities, but the semantic-change problem remains a limitation.
- Paper’s approach: This paper minimizes the domain gap at its source by generating highly realistic synthetic data rather than adapting data or learning domain-resistant features.The approach uses procedural generation to create diverse synthetic individuals without manual intervention.
3. Synthesizing face images
The framework procedurally generates diverse synthetic faces by combining a parametric 3D face model with independently sampled, high-quality assets, expressions, and realistic rendering conditions.
- 3.1. 3D face model: Procedural generation samples diverse 3D face identities and independently combines them with plausible hair, clothing, accessory, and expression assets.The generative model captures population-level shape variation, while independent asset sampling creates varied synthetic individuals without manual intervention.
- 3.1. 3D face model: The face model represents identity, expression, and skeletal pose using a blendshape-based mesh with 7,667 vertices, 7,414 polygons, and four joints.Identity and expression are represented through parameterized displacements, while linear blend skinning applies local joint rotations.
- 3.2. Expression: 27,000 fitted expression parameters plus a manually animated sequence provide varied facial motion, supplemented by random eye gaze and procedural eyelid posing.The manually animated sequence fills expression types that sparse landmark annotations cannot recover, such as cheek puffs.
- 3.3. Texture: High-resolution scan textures, coarse and meso displacement, physically based skin shading, and makeup effects preserve close-up facial realism.The texture collection contains 200 sets of 8192×8192-pixel textures, including albedo and two displacement maps.
- 3.3. Texture: Strand-level hair rendering uses over 100,000 strands per full head and randomly combines 512 scalp styles, 162 eyebrows, 142 beards, and 42 eyelash sets.A physically based procedural shader models hair material properties, including melanin, grayness, dye, and bleaching.
- 3.4. Rendering: Images are rendered with Cycles using randomized camera parameters, depth of field, clothing and accessories, and one of 448 HDRI environments.The framework also generates accurate ground-truth labels alongside color images for landmark, segmentation, and other face-related tasks.
4. Face analysis
The paper evaluates synthetic-only training for face parsing and landmark localization on real datasets, using a common synthetic dataset and targeted label adaptation and augmentation. Across these tasks, synthetic training achieves performance comparable to real-data or state-of-the-art systems, while ablations identify important design choices and limitations.
- Evaluation: Synthetic-only models are evaluated cross-dataset: trained purely on synthetic images and tested on real data, unlike within-dataset state-of-the-art methods.The evaluation covers face parsing and landmark localization.
- Face parsing: A UNet with a ResNet-18 encoder treats face parsing as image-to-image translation and minimizes binary cross-entropy on synthetic labels.The predicted label image has one channel per class and matches the input spatial dimensions.
- Face parsing: Label adaptation transforms synthetic-network predictions toward real-dataset label distributions, addressing systematic differences such as nose boundaries and hair granularity.The adaptation stage is itself an image-to-image translation network.
- Results: Synthetic training achieves results similar to previous within-dataset methods and comparable results to training with real data.This pattern is reported for face parsing and landmark localization, although the synthetic models do not outperform the state of the art in face parsing.
- Ablation studies: Full augmentation is important: without it, synthetic data does not outperform real data, while label adaptation reduces error for synthetic-trained models.Adding label adaptation changes little for real-trained models because their within-dataset labels are already consistent.
- Ablation studies: Removing clothing and hair harms landmark accuracy, supporting the contribution of the hair library and digital wardrobe to data realism.The framework also supports dense landmark regression, including 679 coordinates instead of 68.