Source-linked AI summary
StyleRig: Rigging StyleGAN for 3D Control over Portrait Images
Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, Christian Theobalt
TL;DR
StyleGAN offers photorealistic portraits but lacks interpretable 3D semantic control, whereas 3D morphable models provide such control without comparable portrait photorealism or scene coverage. StyleRig trains RigNet to map 3DMM parameters to a fixed StyleGAN, producing photorealistic images with explicit control over pose, expressions, illumination, and related parameters. The method supports interactive editing and conditional synthesis, while remaining limited by reconstruction quality and preservation of uncontrolled scene content.
Problem
StyleGAN lacks semantically interpretable 3D control, while 3D morphable models lack photorealism and do not model hair, mouth interiors, or backgrounds.
Method
StyleRig trains a RigNet between 3DMM semantic parameters and the input of a fixed pretrained StyleGAN using self-supervised image-domain cycle consistency and differentiable rendering.
Results
StyleRig generates StyleGAN-photorealistic portraits with explicit semantic control and demonstrates interactive editing, style mixing, and conditional image generation.
Takeaways & Limitations
The approach combines computer-graphics-style face control with deep generative models for portrait editing and provides insights into StyleGAN’s learned representations.
Takeaways & Limitations
StyleRig cannot exploit the full expressivity of the face model, control fine-scale details, or explicitly preserve uncontrolled background and hair content during edits.
Abstract
from arXiv · showhide
StyleGAN generates photorealistic portrait images of faces with eyes, teeth, hair and context (neck, shoulders, background), but lacks a rig-like control over semantic face parameters that are interpretable in 3D, such as face pose, expressions, and scene illumination. Three-dimensional morphable face models (3DMMs) on the other hand offer control over the semantic parameters, but lack photorealism when rendered and only model the face interior, not other parts of a portrait image (hair, mouth interior, background). We present the first method to provide a face rig-like control over a pretrained and fixed StyleGAN via a 3DMM. A new rigging network, RigNet is trained between the 3DMM's semantic parameters and StyleGAN's input. The network is trained in a self-supervised manner, without the need for manual annotations. At test time, our method generates portrait images with the photorealism of StyleGAN and provides explicit control over the 3D semantic parameters of the face.
1. Introduction
Existing face models offer interpretable semantic control but limited photorealism and coverage, while StyleGAN produces photorealistic faces whose latent attributes remain entangled. StyleRig combines these complementary strengths through semantic rigging of StyleGAN.
- Motivation: 3D morphable face models provide artist-friendly control over geometry, expressions, reflectance, and illumination, but often lack photorealistic rendering.Their training data can also limit diversity and generalization.
- Motivation: Face-modeling approaches often omit mouth interiors, hair, eyes, and backgrounds because they primarily model the face itself.
- Motivation: StyleGAN’s latent controls entangle attributes such as identity and head pose, preventing explicit rig-like 3D control over the generative model.
- Contribution: StyleRig combines the parametric control of morphable models with StyleGAN’s photorealism using self-supervised two-way cycle consistency and differentiable rendering.The method uses a fixed pretrained StyleGAN and requires no additional data for training.
- Motivation: StyleGAN synthesizes photorealistic faces and exposes coarse, medium, and fine attribute controls, but these attributes are not semantically well defined.
3. Overview
StyleRig maps semantic 3D face parameters to modified StyleGAN latent codes, using a parametric face model and synthetically generated training pairs. It provides explicit control while retaining StyleGAN image quality.
- Overview: StyleRig treats StyleGAN as a mapping from latent code w to a realistic portrait image, then adds semantic control over pose, expression, and illumination.
- Semantic Rig Parameters: The semantic parameter vector includes shape, reflectance, expression, illumination, rotation, and translation, with total dimensionality f = 257.
- Semantic Rig Parameters: Shape and reflectance use PCA models from 200 face scans, while expression uses a PCA model transferred from blendshapes.The shape, appearance, and expression models cover more than 99% of the original variance.
- Training Corpus: Training samples consist of N = 200k latent codes and their StyleGAN-generated images, with latent-vector mixing across resolutions.The resulting pairs support self-supervised training without additional image data or manual annotations.
6. Network Architecture
StyleRig uses a differentiable face reconstruction network and RigNet to convert semantic edits into modified StyleGAN latent codes. Image-domain losses train this mapping while pretrained StyleGAN and reconstruction components remain fixed.
- RigNet: Given latent code w and semantic parameters p, RigNet outputs a modified latent code ˆw whose StyleGAN image should obey p.Separate RigNet networks handle pose, expressions, and illumination.
- Differentiable Face Reconstruction: A pretrained differentiable face reconstruction network maps latent code w to semantic parameters and renders a corresponding synthetic face image.The reconstruction network is trained self-supervised and then fixed.
- Differentiable Face Reconstruction: The reconstruction loss combines dense photometric alignment with sparse landmark alignment, using a rendered-face mask for image comparison.
- Training: StyleRig trains with pretrained, fixed DFR and StyleGAN networks while only RigNet is trainable, defining consistency and edit losses in the image domain.
- Network Architecture: The RigNet encoder independently compresses each 512-dimensional latent sub-vector into 32 dimensions across 18 resolutions.
7. Self-supervised Training
RigNet is trained without manual edit annotations by combining reconstruction, editing, and consistency objectives with cycle-consistent self-supervision. The procedure transfers selected semantic parameters between latent codes while preserving unmodified attributes.
- Self-Supervised Objective: The training objective combines reconstruction, editing, and consistency losses because ground-truth modified images are unavailable.The authors use self-supervision based on cycle-consistent editing and consistency losses.
- Reconstruction Loss: The reconstruction constraint trains RigNet to reproduce each training latent code and anchors the mapping to the correct location in StyleGAN’s latent space.Removing this constraint is reported to degrade image quality, while the pretrained face reconstruction network preserves the control-space semantics.
- Cycle-Consistent Editing: RigNet injects selected semantic parameters from one latent code into another to edit pose, expressions, or illumination.The edited code is mapped back to parameters and evaluated through differentiable rerendering.
- Consistency Loss: Consistency loss keeps parameters that should remain unchanged after editing equal to their original values through a rerendering loss.The original parameter vector is combined with the regressed parameters for the unmodified attributes before rerendering.
- Siamese Training: A Siamese network applies the same editing operation in reverse, producing a two-way cycle-consistency loss from shared-weight towers.The reverse operation injects the original parameters from w into v after injecting v’s parameters into w.
8. Results
StyleRig provides explicit control over StyleGAN portraits through pose, expression, and illumination parameters, demonstrated through mixing, interactive editing, conditional generation, and baseline comparisons.
- StyleRig controls pose, expression, and illumination in StyleGAN-generated images through three applications: style mixing, interactive rig control, and conditional image generation.
- Style Mixing: StyleRig transfers semantic parameters from source to target images while preserving target identity and scene context such as background, hair, and accessories.
- Style Mixing: StyleGAN’s coarse latent vectors mainly control rotation, coarse and medium vectors control expression, and medium-resolution vectors mostly control light direction.
- Interactive Rig Control: Users can interactively change pose, expression, and scene illumination through a face mesh, with RigNet generating updated images at approximately 5 fps.
- Conditional Image Generation: Fixing pose, expression, or illumination inputs converts pretrained StyleGAN into a conditional generator, requiring less than 24 hours of training versus more than 41 days for StyleGAN from scratch.
- Comparisons to Baseline Approaches: The full approach obtains the highest quality among the reported baseline comparisons, while removing reconstruction or consistency losses degrades latent validity or parameter consistency.
9. Limitations
StyleRig does not exploit the full expressivity of the parametric face model and has additional limitations from reconstruction quality and unconstrained scene context.
- StyleRig cannot transfer all modes of parametric control, including in-plane rotation and many facial expressions, because these modes are underrepresented in StyleGAN’s training data.
- The differentiable face reconstruction network cannot reconstruct fine-scale detail, limiting explicit control over those details.
- Background and hairstyle are not explicitly constrained by the parametric face model, so they may change when editing controlled parameters.
10. Conclusion
StyleRig provides face rig-like control over a pretrained and fixed StyleGAN through self-supervised training, without additional images or manual annotations. It combines explicit semantic control with StyleGAN photorealism at test time.
- StyleRig provides face rig-like control over a pretrained and fixed StyleGAN network.
- The network is trained self-supervised and requires neither additional images nor manual annotations.
- At test time, StyleRig generates photorealistic face images while providing explicit control over semantic parameters.
–Supplementary Material–
The supplementary material provides further training details and evaluations, and directs readers to the supplementary video for additional editing results.
- The supplemental document provides further training details and evaluations.
- The authors strongly recommend watching the supplementary video for more editing results.
1. Training Details
Training uses parameter-specific loss weights for pose, expression, and illumination, while face translation is not edited because landmark-based cropping correlates rotation and translation.
- Pose, expression, and illumination editing use λland values of 17.5, 100.0, and 7.8, respectively.The same hyperparameters are used for both editing and consistency losses.
- Simultaneous control weights rotation, expression, and illumination losses by 1.0, 1000.0, and 0.001, respectively.Editing and consistency losses receive equal weights.
- Face translation is not edited because landmark-based cropping creates a strong correlation between head rotation and translation.The face is allowed to translate while rotating rather than preserving translation.
2. Evaluation of Simultaneous Parameter Edits
A single network can simultaneously edit pose, expression, and illumination, while evaluations characterize quality and consistency trade-offs relative to parameter-specific networks. StyleRig also supports identity transfer, expression transfer, and editing real images, with artifacts possible after latent inversion.
- Simultaneous Parameter Edits: A single network can simultaneously control pose, expression, and illumination, producing high-quality results.
- Simultaneous Parameter Edits: Simultaneous control reproduces target pose less accurately than pose-only control as rotation edits become larger.The simultaneous network’s rotation editing loss increases faster with edit magnitude.
- Simultaneous Parameter Edits: Expression-only control better preserves pose, illumination, and identity than simultaneous control, despite similar expression editing loss.Its consistency losses are lower than those of the simultaneous network for expressions.
- Identity Geometry Transfer: StyleRig transfers source identity geometry to target images using the identity component of the 3DMM.
- Expression Transfer: Compared with ELEGANT, StyleRig produces higher-quality expression transfers and better transfers of source expressions.
- Real-Image Editing: StyleRig edits real images after StyleGAN latent optimization, but optimized embeddings far from training data can produce artifacts.
4. Comparison
StyleRig produces higher-quality expression transfers than ELEGANT with fewer artifacts, while extending to real-image editing and revealing important control limitations.
- StyleRig transfers source expressions to target images with higher quality and fewer artifacts than ELEGANT.The comparison also reports better transfer of the source expressions.
- StyleRig can edit real images by using Image2StyleGAN to embed the image before RigNet computes an edited embedding.The approach supports editing high-resolution images, but optimized embeddings can fall outside RigNet’s training distribution and produce artifacts.
- In-plane rotations cannot be produced by the approach.This failure is attributed to bias in the training data used for StyleGAN.
- Expressions beyond mouth opening or smiling are often ignored or incorrectly mapped.The paper attributes these failures to bias in StyleGAN’s training data.
- High-frequency image details cannot be controlled because the differentiable face reconstruction network reconstructs only coarse geometry and appearance.