PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffusionRig: Learning Personalized Priors for Facial Appearance Editing

Zheng Ding, Xuaner Zhang, Zhihao Xia, Lars Jebe, Zhuowen Tu, Xiuming Zhang

arXiv:2304.06711v1cs.CVcs.GR

TL;DR

DiffusionRig addresses personalized facial editing that changes lighting, expression, or pose while preserving identity and high-frequency details. It combines physical 3D guidance with generic and person-specific diffusion priors, and the paper reports identity-preserving editing under these conditions. The authors also identify limits tied to small-personal-dataset fine-tuning, DECA estimates, and dramatic pose changes.

  • Problem

    Photorealistically changing portrait lighting, expression, or pose while preserving identity and high-frequency facial details is challenging because generic priors often miss person-specific characteristics.

    Source: §1 · pp. 1–2

  • Method

    DiffusionRig uses DECA-derived physical buffers and a global latent code in a diffusion model, first learning generic priors and then fine-tuning the denoiser on one person’s photos.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    The paper reports that personalized priors enable facial appearance editing while preserving identity and high-frequency details.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    DiffusionRig controls facial synthesis using explicit physical-buffer conditions and implicit global-latent conditioning while learning personalized priors on top of generic ones.

    Source: §6 · p. 8

    Fine-tuning on small portrait datasets limits scalability, dramatic pose changes may require out-of-scope background inpainting, and DECA estimates can be inaccurate for extreme expressions or couple lighting with skin tone.

    Source: §6 · p. 8

Abstract

from arXiv · show

We address the problem of learning person-specific facial priors from a small number (e.g., 20) of portrait photos of the same person. This enables us to edit this specific person's facial appearance, such as expression and lighting, while preserving their identity and high-frequency facial details. Key to our approach, which we dub DiffusionRig, is a diffusion model conditioned on, or "rigged by," crude 3D face models estimated from single in-the-wild images by an off-the-shelf estimator. On a high level, DiffusionRig learns to map simplistic renderings of 3D face models to realistic photos of a given person. Specifically, DiffusionRig is trained in two stages: It first learns generic facial priors from a large-scale face dataset and then person-specific priors from a small portrait photo collection of the person of interest. By learning the CGI-to-photo mapping with such personalized priors, DiffusionRig can "rig" the lighting, facial expression, head pose, etc. of a portrait photo, conditioned only on coarse 3D models while preserving this person's identity and other high-frequency characteristics. Qualitative and quantitative experiments show that DiffusionRig outperforms existing approaches in both identity preservation and photorealism. Please see the project website: https://diffusionrig.github.io for the supplemental material, video, code, and data.

1. Introduction

Photorealistically editing lighting, expression, and pose while preserving identity and fine facial details remains challenging because generic priors often miss person-specific features. DiffusionRig combines coarse 3D guidance with generic and personalized diffusion priors to produce edited portraits of a specific person.

1. Introduction

  • Generic facial priors often miss a test identity’s high-frequency characteristics, even though personal photos are often available for learning person-specific priors.The approach uses roughly 20 photos to adapt generic priors to the person of interest.Source: §1 · p. 2
  • DECA provides editable 3DMM parameters, but its coarse geometry, restrictive reflectance and lighting assumptions, and missing hairstyle or accessories make direct renderings look like CGI.Changing 3DMM parameters nevertheless offers a useful way to rig expression, pose, and lighting.Source: §1 · p. 2
  • DiffusionRig edits a 3DMM’s appearance and uses a diffusion model to map edited physical buffers to photorealistic images.The buffers include surface normals, albedo, and Lambertian rendering.Source: §1 · p. 2
  • The two-stage approach first learns generic facial priors, then fine-tunes on roughly 20 images of one person to preserve identity and high-frequency details during editing.The generic model alone offers no identity-preservation guarantee because coarse physical buffers lack sufficient identity information.Source: §1 · p. 2

2. Related Work

Related work spans generative models, 3D face representations, and personalized priors. DiffusionRig uses pixel-aligned physical conditions and personal fine-tuning to support controllable editing with a smaller personal dataset than MyStyle requires.

Generative Modeling

  • Denoising diffusion models learn to turn random-noise images into photorealistic images; this work conditions its model on pixel-aligned surface-normal, albedo, and Lambertian-rendering images, while Diff-AE learns facial-attribute latent spaces via autoencoding [34].Source: §2 · p. 2

Facial Appearance Modeling

  • 3DMMs provide a parameter space for facial characteristics, while image-space estimates can represent hair and accessories but lack physically meaningful rigging parameters [39].DECA predicts albedo and spherical-harmonics lighting from a single face image [9].Source: §2 · p. 3
  • 3DMM geometry, albedo, and lighting remain coarse, motivating fine-grained capture setups and implicit appearance models to bridge the realism gap [48] [50].The cited implicit models aim to avoid explicitly describing precise reflectance and lighting [12] [28] [30] [35] [36] [42] [52] [53].Source: §2 · p. 3

Personalized Priors

  • Earlier personalized-prior work includes exemplar imagery, supplemental attributes, identity penalties, and facial-component dictionaries [49] [25].The cited approaches do not explicitly learn personalized priors, and unbiased identity scoring remains challenging.Source: §2 · p. 3
  • MyStyle fine-tunes StyleGAN for a specific identity but does not provide precise 3D rigging and requires a larger personal dataset than DiffusionRig [32].DiffusionRig uses continuous physical-space conditions for controllable editing.Source: §2 · p. 3

3. Preliminaries

The preliminaries introduce 3D morphable face models as a compact, editable facial representation and diffusion models as a progressive denoising process. Together, they provide the physical controls and image-generation mechanism used by the paper.

3.1. 3D Morphable Face Models

  • 3DMMs use compact latent spaces to represent pose, geometry, and expression; this paper uses FLAME, while DECA adds albedo and spherical-harmonics lighting estimation from a portrait [9].Editing FLAME parameters, albedo, or lighting supports rigging, but DECA renderings have a substantial realism gap from photos.Source: §3.1 · p. 3

3.2. Denoising Diffusion Probabilistic Models

  • DDPMs progressively denoise random noise images to generate photorealistic images, reversing a process that gradually adds noise.The overview describes the model as using a denoising network to predict noise at each timestep.Source: §3.2 · pp. 3–4
  • During DDPM training, the denoising network predicts the noise in a timestep-corrupted image, and generation repeatedly denoises random input.The predicted noise is compared with the actual noise in the loss.Source: §3.2 · p. 4

4. Method

DiffusionRig learns to generate facial images under physical appearance conditions while limiting identity shift through person-specific fine-tuning. It combines pixel-aligned physical buffers, a global latent code, and a two-stage training process.

4. Method

  • The method first learns generic facial priors from portraits conditioned on estimated appearance buffers, then fine-tunes on one person’s photos to learn personalized priors.The personalized stage is intended to prevent identity shift during appearance editing.Source: §4 · p. 4

4.1. Learning Generic Face Priors

  • DECA-derived surface normals, albedo, and Lambertian renderings provide pixel-aligned physical conditions for diffusion-based synthesis and physically meaningful editing.The buffers are coarse and not photorealistic, but changing DECA parameters rigs the model’s output.Source: §4.1 · p. 4
  • A global latent code complements physical buffers by encoding appearance information they do not model, such as hair, hats, glasses, and background.The denoiser receives the noisy image and physical buffers, while the encoder extracts the global code from the original image.Source: §4.1 · p. 4

4.2. Learning Personalized Priors

  • For personalized fine-tuning, the global encoder is fixed while the denoising model adapts to about 20 photos; FLAME shape is averaged across the album to reduce sensitivity to extreme poses or expressions.The mean-shape procedure assumes the person’s general facial shape does not change drastically over a reasonable period.Source: §4.2 · p. 5

4.3. Model Architecture

  • The denoiser is ADM-based and accepts a global latent code through layer-wise feature scaling and shifting, while a ResNet-18 serves as the encoder.Training uses a P2-weighted noise-prediction loss, which the authors report speeds training and produces high-quality images compared with constant weighting.Source: §4.3 · p. 5

4.4. Implementation Details

  • Stage 1 uses 70,000 FFHQ images for 50,000 iterations; Stage 2 fine-tunes on 10–20 images per person for 5,000 iterations.Stage 1 takes around 15 hours on eight A100 GPUs, and Stage 2 completes within 30 minutes on one V100 GPU.Source: §4.4 · p. 5

5. Experiments

The experiments test physical-buffer editing, global-latent control, personalized-model identity transfer, and quantitative evaluation and ablations. They report identity-preserving edits and show how the physical buffers, latent code, and personalized weights affect generated portraits.

5.1. Rigging Appearance With Physical Buffers

  • DiffusionRig makes appearance edits while preserving identity; GIF shifts identity, while MyStyle struggles with dramatic expression or pose changes and does not trivially support relighting.The comparison uses 20 images per identity for DiffusionRig.Source: §5.1 · pp. 5–6, Figure 4 · p. 6

5.2. Rigging Appearance With Global Latent Code

  • Changing the global latent code alters background, glasses, and hairstyle, while physical buffers carry head pose and expression information.In the Figure 5 comparison, latent codes vary across columns and physical buffers across rows.Source: §5.2 · p. 6

5.3. Identity Transfer With Learned Priors

  • With physical buffers and latent code held constant, swapping personalized models changes the generated identity, supporting that the models learn person-specific priors.The Figure 6 comparison includes “no-swap” results using each person’s own model.Source: §5.3 · pp. 6–7, Figure 6 · p. 7

5.4. Baseline Comparisons & Evaluation Metrics

  • The evaluation covers rigging quality, identity preservation, and photorealism, using DECA re-inference error, face re-identification error, and a user study.The user study presents generated expression- or pose-edited images alongside originals and asks whether the generated image is a real image of the same person.Source: §5.4 · pp. 7–8
  • Both MyStyle and DiffusionRig preserve identity across 400 expression-edited images each for Obama and Swift, while DiffusionRig preserves identity better for dramatic head-pose changes.The automatic face re-identification metric can reward making no edit; Figure 4 and Table 1 are cited as evidence DiffusionRig avoids that degenerate solution.Source: §5.4 · p. 7, Table 2 · p. 8

5.5. Ablation Study

  • Without Stage 2 personalized priors, the model uses physical buffers for pose, expression, and lighting but cannot preserve the person’s identity during editing.Source: §5.5 · p. 8
  • Using 1, 5, or 10 Stage 2 images yields worse results than using 20, with more images capturing high-frequency facial characteristics such as wrinkles.The figure caption notes that fewer images may yield blurry results and make edits difficult to rig to new conditions.Source: §5.5 · p. 8, Figure 7 · p. 8
  • Pixel-aligned physical buffers rig appearance more accurately than vector or feature conditioning, whose outputs fail to follow the desired physical guidance.Vector conditioning concatenates DECA parameters to the global latent code; feature conditioning uses physical buffers as non-spatial encoder features.Source: §5.5 · p. 8, Figure 8 · p. 8

6. Limitations & Conclusion

DiffusionRig combines generic and personalized priors with physical-buffer and latent-code conditioning, while remaining constrained by its training data and DECA estimates.

Limitations

  • A small portrait dataset for finetuning limits DiffusionRig’s scalability for massive user adoption.This is the stated data-scope limitation.Source: §6 · p. 8
  • Dramatic head-pose changes can reveal previously occluded regions, requiring background inpainting beyond this paper’s scope.The limitation applies when an edit involves a dramatic pose change.Source: §6 · p. 8
  • DECA’s limited estimates can miss extreme expressions and sometimes couple estimated lighting with skin tone.DiffusionRig relies on DECA to obtain its physical buffers.Source: §6 · p. 8

Conclusion

  • The method learns generic face priors before personalized priors, using physical buffers and a global latent code to drive facial image synthesis.The conclusion characterizes these as explicit and implicit conditioning, respectively.Source: §6 · p. 8

A. More Results on Personalized Editing

Supplementary results show physical buffers used to edit portrait images across facial expression, lighting, and head pose.

A. More Results on Personalized Editing

B. Extreme Lighting Editing

The supplement describes an inference-time lighting representation for extreme lighting examples and gives details about the personal photo collections used for Stage 2.

B. Extreme Lighting Editing

  • At inference, a different lighting representation can model directional lighting with cast shadows through ray casting.The supplement shows directional and RGB lighting examples; the model’s regression in those examples is described only as slight.Source: B. Extreme Lighting Editing · p. 12, Figure S2 · p. 12

C. Personal Photo Collections

  • Stage 2 training uses internet-crawled celebrity photos and everyday photos for non-celebrities.Figure S3 shows collections for Taylor Swift and a non-celebrity.Source: C. Personal Photo Collections · p. 12
  • With 20 images, MyStyle cannot learn personalized priors well, whereas its finetuning requirement is 92–279 images.The comparison is made against DiffusionRig’s use of 20 images.Source: C. Personal Photo Collections · p. 12

D. Neural Network Architecture Details

The supplement describes the model’s global-code conditioning and reports how physical-buffer choices affect facial detail and lighting accuracy.

D. Neural Network Architecture Details

  • The global encoder modifies ResNet-18 to output a latent code through a feature-extraction layer.Source: D. Neural Network Architecture Details · p. 12
  • The diffusion model adds a global latent code as a condition alongside physical buffers and the noise image.The code scales and shifts model features; the architecture details cover 256 × 256 and 512 × 512 models.Source: D. Neural Network Architecture Details · p. 12, Table S1 · p. 13
  • Figure S3 shows Stage 2 training collections for Taylor Swift and a non-celebrity.The supplement says celebrity photos are crawled from the internet and non-celebrity collections use everyday photos.Source: Figure S3 · p. 13

E. Different Types of Pixel-Aligned Buffers

  • Normals, albedo, and Lambertian rendering preserve details such as the mouth better, while N+S cannot render accurate lighting because albedo is missing.Figure S4 ablates different input conditions; N denotes normals, S spherical-harmonics layers, L Lambertian rendering, and A albedo.Source: E. Different Types of Pixel-Aligned… · p. 13, Figure S4 · p. 13

F. Higher-Resolution Results (512 × 512)

DiffusionRig is trained at 512 × 512 resolution, with results shown for two new celebrities.

F. Higher-Resolution Results (512 × 512)

Loading 2304.06711v1…