Source-linked AI summary
Neural Face Editing with Intrinsic Image Disentangling
Zhixin Shu, Ersin Yumer, Sunil Hadap, Kalyan Sunkavalli, Eli Shechtman, Dimitris Samaras
TL;DR
Traditional face editing relies on costly, task-specific pipelines for reconstructing and manipulating facial attributes. This paper proposes an end-to-end adversarial network with physically grounded disentangled representations, enabling semantic edits from single in-the-wild images while constraining unaffected properties.
Problem
Existing face-editing methods often require costly, task-specific reconstruction algorithms and richer data or strong priors for in-the-wild images.
Method
The network infers disentangled surface normals, albedo, lighting, and a matte layer, using differentiable in-network image formation and additional intrinsic-property constraints.
Results
The learned low-dimensional intrinsic-property manifolds support semantic edits from a single image, including relighting, expression, appearance, eyewear, and aging, with more realistic results when unaffected properties are constrained.
Takeaways & Limitations
Physically meaningful latent variables let users manipulate one facial-appearance factor while keeping orthogonal properties fixed for realistic editing.
Takeaways & Limitations
The current face masks exclude hair, limiting control over edits such as aging that affect hair.
Abstract
from arXiv · showhide
Traditional face editing methods often require a number of sophisticated and task specific algorithms to be applied one after the other --- a process that is tedious, fragile, and computationally intensive. In this paper, we propose an end-to-end generative adversarial network that infers a face-specific disentangled representation of intrinsic face properties, including shape (i.e. normals), albedo, and lighting, and an alpha matte. We show that this network can be trained on "in-the-wild" images by incorporating an in-network physically-based image formation module and appropriate loss functions. Our disentangling latent representation allows for semantically relevant edits, where one aspect of facial appearance can be manipulated while keeping orthogonal properties fixed, and we demonstrate its use for a number of facial editing applications.
1. Introduction
The paper targets general face editing in-the-wild with an end-to-end network that disentangles intrinsic facial properties and supports semantic manipulation while preserving unrelated properties.
- Motivation: Prior face-editing pipelines reconstruct geometry, texture, and illumination with task-specific methods that are costly and poorly generalize to in-the-wild variation.These methods often require richer data or strong task-specific reconstruction priors.
- Motivation: The proposed network learns a compact facial-appearance manifold covering viewpoint, lighting, expression, facial hair, and age without expensive data capture.The goal extends beyond the narrower manipulations represented by earlier morphable face models.
- Approach: The model explicitly infers surface normals, diffuse albedo, and illumination, then reconstructs images through an in-network forward rendering model.Training uses geometry, albedo, and lighting priors together with adversarial image-reconstruction supervision and weak intrinsic-property supervision.
- Approach: A matte layer separates face foreground from arbitrary backgrounds, enabling face-specific reconstruction pathways without distorting background reconstruction.This foreground-background separation is designed for natural images with unconstrained backgrounds.
- Results: Low-dimensional embeddings for intrinsic properties enable single-image edits such as relighting, smiling, aging, and appearance changes while irrelevant properties remain constrained.The paper reports significantly more realistic results than other learning-based face-editing approaches when unaffected physical properties are constrained.
- Contributions: The main contributions are an end-to-end face-editing generative network, in-network physically based image formation, and statistical losses such as batchwise white shading.These components are intended to improve disentanglement of shape, illumination, and albedo in latent representations.
2. Related Work
The paper positions its method against task-specific face-editing systems and builds a general face manifold using physically grounded, differentiable rendering with explicit or implicit representations.
- Related Work: Earlier face-manipulation methods support tasks such as relighting, attribute editing, expression editing, performance authoring, and aging but are often application-specific and difficult to generalize.This paper instead aims to learn a general facial-appearance manifold supporting these tasks.
- Related Work: Intrinsic-image methods decompose images into physical components, while this work learns a face-specific appearance model with weak supervision under a similar rendering model.The related rendering assumptions include Lambertian reflectance, low-frequency illumination, and priors on geometry, albedo, and illumination.
- Method: The method formulates face generation as an end-to-end network with explicit in-network shape, albedo, lighting, image formation, and matting components.The architecture introduces shading and image-formation layers, alternative face representations, and in-network matting for arbitrary backgrounds.
- Method: Differentiable rendering layers convert albedo, normals, and lighting into image-space appearance, producing disentangled latent variables for physically meaningful factors.The latent variables separately encode albedo, normals, and lighting, allowing edits along one factor's manifold while others remain fixed.
- Rendering: Spherical harmonics provide a 9-dimensional lighting representation that makes the Lambertian shading function analytical and differentiable under distant illumination.This assumption enables the shading module to operate inside the network.
- Representations: The explicit representation models per-pixel normal and albedo maps, whereas the implicit representation models face-space coordinates with normal and albedo maps aligned in UV space.The implicit alternative is introduced because pose and expression can jointly change explicit image-space normals and albedo.
3. Implementation
The implementation uses an encoder–decoder generative network with separate latent representations for face properties, background, and matte masks. Training combines image reconstruction and adversarial objectives with intermediate constraints that guide physically meaningful disentangling from in-the-wild images.
- 3.1. Network Architecture: The encoder uses three convolutional layers with 32, 64, and 64 filters, followed by max-pooling and ReLU nonlinearities.For 3 × 64 × 64 inputs, padding produces a 64 × 8 × 8 final encoder representation.
- 3.1. Network Architecture: A shared 128-dimensional latent representation connects to separate background, mask, lighting, and foreground-property representations.Lighting is represented directly by a 27-dimensional vector containing concatenated RGB spherical-harmonics coefficients.
- 3.2. Training: The main reconstruction objective compares input image Ii with output Io using Erecon = ||Ii − Io||2, while an energy-based discriminator supplies adversarial supervision.The discriminator is an autoencoder trained with a margin-based reconstruction error, and the generator’s adversarial term is D(I′).
- 3.2. Training: Because reconstruction and adversarial losses alone can produce semantically meaningless latents, intermediate losses constrain physically encoded rendering elements.The paper illustrates a degenerate solution in which albedo captures shading while shading remains constant.
- 3.2. Training: A Morphable Model provides pseudo-ground-truth normals and lighting targets, with L2 losses used for normal-map and lighting estimation.Lighting targets are computed from estimated normals and the input image using least-squares optimization with a constant-albedo assumption.
- 3.2. Training: Retinex-inspired smoothness losses regularize albedo and shading, while batch-wise white shading constrains shading magnitude and cross-channel color balance.The BWS implementation uses red, green, and blue shading channels over m pixels, with c = 0.75 in all experiments.
4. Experiments
Experiments show that the network learns disentangled face representations that support robust reconstruction, semantic edits, and lighting transfer while preserving unrelated appearance factors.
- Training setup: CelebA images are fitted with a 3D Morphable Model to provide rough normal and lighting estimates used only for training losses.
- Baseline comparisons: Even with a 128-dimensional bottleneck versus the baseline’s 265 dimensions, the network reconstructs faces more robustly under complex backgrounds, pose, illumination, and occlusion.The baseline uses an otherwise matching auto-encoder architecture and has slightly greater capacity.
- Baseline comparisons: The model explicitly exposes albedo, shading, and normal maps, and its inferred normals extend beyond the 16-identity morphable-model subspace while retaining more input identity information.Weak morphable-model supervision is combined with appearance-based rendering loss to improve shape fitting.
- Face editing by manifold traversal: Traversing albedo and normal manifolds produces smiles with expression and teeth changes while preserving other image aspects, outperforming baseline latent-space traversal.The baseline more poorly reconstructs pose and identity and captures the smiling transformation less effectively.
- Face editing by manifold traversal: Implicit UV and normal traversals capture smiling more directly, whereas implicit-albedo traversal produces appearance artifacts and an incorrect frontal pose.The UV representation separates spatially overlapping lips, mouth, and teeth regions.
- Face editing by manifold traversal: Progressive traversal strengthens smiling features, aging traversal changes face shape and texture while retaining pose, lighting, and eyewear, and the same network supports both edits.The demonstrated semantic attributes include expression and age, with traversal controlled by λ.
- Limitations: The current face masks exclude hair, limiting control over edits such as aging that inherently affect hair.The paper states that masks including hair could address this limitation.
- Relighting: Lighting transfer generates target shading under source illumination and renders it with target appearance factors, transferring lighting across disparate identities, genders, and poses.The method estimates factors for source and target images and uses target normals with source lighting.
5. Conclusions
The paper proposes a physically grounded rendering-based disentangling network for faces, enabling realistic editing through constraints applied during manipulation.
- The network is specifically designed to disentangle face appearance using physically grounded rendering.
- Disentangling enables realistic face editing by allowing trivial constraints at manipulation time.
- The approach supports faces in the wild with real, arbitrary backgrounds through in-network rendering.
A. Implementation: more details
This section gives implementation details for the rendering layers used by the network.
- The section details implementation of the shading layer.
- The section details implementation of the image-formation layer.
- These rendering layers are denoted fshading and fimage-formation in the paper.
A.1. Shading Layer
The shading layer represents illumination with spherical harmonics, with forward and backward processes described through the paper’s equations.
- The shading layer uses a spherical harmonics representation of illumination.
- The forward shading process is described by equations (3), (4), and (5) in the main paper.
- The section provides the backward process for shading, including partial derivatives with respect to the relevant quantities.
A.2. Image Formation Layer
For the foreground, the image-formation layer computes a per-element product, and its backward process is obtained through corresponding partial derivatives.
- The foreground image-formation layer performs a per-element product in the forward process.
- This forward operation is referenced as equation (2) in the main paper.
- The section derives partial derivatives for the backward process of foreground image formation.
B. Quantitative Experiments
The network is evaluated for illumination estimation and normal reconstruction against controlled-data references and a direct 3DMM fit. Its illumination estimates show lower within-condition variance than least-squares 3DMM estimates.
- 0.16 average illumination-coefficient variance was obtained by the network, compared with 0.36 for a 3D Morphable Model using least-square estimation.The evaluation sampled 7,000 Multi-PIE images across identities, poses, and 20 controlled light sources.
- The illumination evaluation used Multi-PIE images with controlled illumination and measured variance of coefficients L_i within each illumination condition.Controlled conditions provide known cross-subject lighting correspondences, although the actual lighting setup is not known.
- Normal reconstruction was compared with ground-truth normals and normals obtained from a direct 3DMM fit using images of five individuals captured with light-stage data.The subjects included two women and three men.
C. Additional Results
Additional experiments demonstrate disentangled face editing across glasses, beards, aging, smiling, and relighting, alongside comparisons for normal and illumination reconstruction. The method edits selected manifolds for targeted changes and produces qualitatively competitive relighting results with fewer artifacts.
- Illumination estimation: The network’s illumination estimates are more stable than least-squares morphable-model estimates under identical controlled lighting.The estimates are represented as spherical harmonics lighting coefficients and exhibit less variance across Multi-PIE face images.
- Reconstruction: Normal reconstruction is compared with ground-truth normals and a 3D morphable-model fit, using separate reconstruction and error views.The figure presents input images, ground-truth normals, the proposed reconstruction, its error, the 3DMM fit, and its error.
- Eye-glasses: Eye-glasses edits use the albedo manifold because glasses affect reflectance without associated geometry or warping.Editing the normal manifold has almost no effect, while editing the UV manifold slightly ages the face; editing all manifolds changes geometry and appearance.
- Attribute editing: Beard, aging, and smiling edits traverse manifolds selected for each attribute, with beard edits restricted to albedo and aging or smiling using combinations of disentangled manifolds.Beard traversals use ZAi with λ = 0.03, 0.02, and 0.01; aging uses ZAi, ZNi, and ZUV; smiling uses ZNi and ZUV.
- Relighting: Relighting transfers source-image lighting to target faces, captures the target lighting over the full face, and produces fewer artifacts than two previous techniques.The comparison is described against spherical-harmonics radiance maps and edge-preserving filters.