Source-linked AI summary
PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering
Yurui Ren, Ge Li, Yuanqi Chen, Thomas H. Li, Shan Liu
TL;DR
Portrait editing needs intuitive, disentangled controls while preserving realistic appearance, but existing methods often lack fine-grained semantic control or rely on motion imitation. PIRenderer uses 3DMM parameters with a mapping, warping, and editing pipeline to control portrait motion. The paper reports accurate photo-realistic editing, motion imitation, and coherent audio-driven videos from a single reference image and audio stream.
Problem
Portrait editing lacks broadly applicable, fine-grained semantic controls for facial motion, while existing approaches often use indirect motion imitation.
Method
PIRenderer uses 3DMM-based motion descriptors and mapping, warping, and editing networks to render controlled portraits while preserving source attributes.
Results
PIRenderer generates photo-realistic portraits with accurate user-specified motions, supports motion imitation, and produces coherent audio-driven videos from one portrait image and driving audio.
Takeaways & Limitations
Semantically disentangled parameters support intuitive portrait editing, motion imitation, and extension to audio-driven facial reenactment.
Abstract
from arXiv · showhide
Generating portrait images by controlling the motions of existing faces is an important task of great consequence to social media industries. For easy use and intuitive control, semantically meaningful and fully disentangled parameters should be used as modifications. However, many existing techniques do not provide such fine-grained controls or use indirect editing methods i.e. mimic motions of other individuals. In this paper, a Portrait Image Neural Renderer (PIRenderer) is proposed to control the face motions with the parameters of three-dimensional morphable face models (3DMMs). The proposed model can generate photo-realistic portrait images with accurate movements according to intuitive modifications. Experiments on both direct and indirect editing tasks demonstrate the superiority of this model. Meanwhile, we further extend this model to tackle the audio-driven facial reenactment task by extracting sequential motions from audio inputs. We show that our model can generate coherent videos with convincing movements from only a single reference image and a driving audio stream. Our source code is available at https://github.com/RenYurui/PIRender.
1 . Introduction
PIRenderer addresses portrait editing by combining semantically meaningful motion controls with neural rendering for realistic face and background synthesis. It targets direct editing, motion imitation, and audio-driven reenactment.
- Portrait editing requires accurate 3D face geometry alongside photo-realistic faces and backgrounds because portrait images are visually demanding.
- Existing motion descriptors with 2D spatial information can improve target-image generation but hinder intuitive portrait editing.
- 3DMM-based parameters provide semantically meaningful, disentangled controls for facial expressions, head rotations, and translations.
- The model supports user-specified editing, motion imitation of another individual, and audio-driven reenactment from a single portrait image and driving audio stream.
- Prior motion-imitation methods often require hours of labeled videos for one person and retraining for each individual.
- PIRenderer maps motion descriptors to latent vectors, warps source images into coarse results, and edits those results into final images.
3 . Our Approach
PIRenderer uses 3DMM-derived motion descriptors and a three-stage neural renderer to generate portrait images with controlled target motions while preserving source attributes. A temporal coefficient window addresses errors in reconstructed motion parameters.
- PIRenderer generates photo-realistic portrait images by modifying target expressions, head poses, and translations while preserving source identity, illumination, and background.
- The architecture combines a Mapping Network, Warping Network, and Editing Network to transform motion descriptors and source portraits into final images.
- 3 .1. Target Motion Descriptor: The motion descriptor uses a subset of 3DMM parameters to provide intuitive and fine-grained semantic control.
- 3 .1. Target Motion Descriptor: 3DMM shape coefficients represent facial identity and expression, while rotation and translation parameters specify the desired face motions.
- 3 .1. Target Motion Descriptor: A temporal window of coefficients is used around each center frame to reduce the effects of noisy 3D face reconstruction estimates.
3 .2. PIRenderer for Semantic Control
PIRenderer maps semantic motion descriptors into latent controls, warps source portraits toward target motions, and edits the coarse results to preserve source information while correcting artifacts.
- Mapping Network: The mapping network converts motion descriptors p into latent vectors z that control subsequent rendering operations.Affine transformations of z produce AdaIN parameters for injecting motion into the warping and editing networks.
- Warping Network: The warping network estimates source-to-target deformations and uses them to create coarse results from the source image.It predicts a flow field of coordinate offsets and warps the source image; the flow is produced at one-quarter input resolution.
- Warping Network: Warping is trained without ground-truth flow fields by minimizing perceptual reconstruction error between warped and target images.The loss uses L1 distances between VGG-19 activation maps across pyramid resolutions.
- Editing Network: The editing network refines warped coarse results using the coarse image, source image, and latent motion vector to generate the final prediction.Skip connections preserve source textures, while AdaIN injects motion information.
3 .3. Extension on Audio-driven Reenactment
The audio-driven extension generates facial motions sequentially from audio and prior motions, using a stochastic conditional normalizing flow to model varied possible outputs.
- Motivation: Audio-driven reenactment generates convincing expressions and poses from driving audio and a source image.Directly mapping audio to realistic imagery or low-level motion descriptors is described as challenging because audio does not fully determine motion.
- Conditional Motion Generation: A stochastic normalizing flow models multiple possible motions for a given audio stream instead of predicting deterministic mean motions.The flow maps samples from a simple latent distribution into the conditional motion distribution.
- Sequential Generation: At time i, the model conditions current motion pi on the previous k motions and the audio window ai−k:i+τ.An LSTM inside the flow captures temporal correlations, and training uses a single negative log-likelihood loss.
4 .1. Implementation Details
The experiments use cropped and resized talking-head videos, while the qualitative comparison contrasts StyleRig with PIRenderer on StyleGAN-generated source images.
- Dataset: The model is trained on VoxCeleb, containing 22496 talking-head videos extracted from YouTube.Faces are cropped into fixed bounding boxes, resized to 256×256, and paired with extracted audio for reenactment experiments.
- Qualitative Comparison: The qualitative intuitive-editing comparison uses StyleGAN-generated images as source images.StyleRig results appear on the left and PIRenderer results on the right, with red arrows marking undesired and corresponding correct modifications.
- Evaluation: Table 1 reports evaluation results for the intuitive portrait editing task.The accompanying comparison discusses realism and motion fidelity using FID, AED, and APD.
4 .2. Intuitive Portrait Image Editing
PIRenderer is evaluated for intuitive portrait editing and motion imitation, where it produces realistic images with accurate movements while preserving source information.
- Intuitive editing: The evaluation compares StyleRig and PIRenderer using 50k editing images per model generated from sampled expressions and rotations.Translations are excluded because StyleRig does not support translation editing.
- Intuitive editing: StyleRig achieves a lower FID, but its higher AED and APD indicate less faithful reconstruction of target motions.StyleRig can weaken motions away from the distribution center and alter motion-irrelevant factors such as glasses and clothing.
- Intuitive editing: PIRenderer generates realistic images with correct global poses, vivid microexpressions, and preserved irrelevant source information.The model remains effective on FFHQ despite not being trained with that dataset.
- Intuitive editing: PIRenderer can directly edit real-world images with fewer artifacts and realistic source textures than the compared approach.The comparison concerns results shown in Fig. 6, where artifacts in some images are marked by red arrows.
- Motion imitation: In motion imitation, FOMM and GFLA perform strongly for same-identity reconstruction but decline on cross-identity motion imitation.Their spatial, subject-specific motion descriptors may help reconstruction while reducing cross-identity reenactment performance.
- Motion imitation: PIRenderer achieves the best fooling scores in both motion-imitation tasks, indicating more realistic generated results.The study uses 200 images per task, with each ground-truth/generated pair compared five times by different MTurk volunteers.
- Motion imitation: Qualitatively, PIRenderer combines realistic results and accurate movements while preserving source identity better under differing face shapes and driving-video occlusions.X2Face shows warping artifacts, Bi-layer oversmooths details, and GFLA and FOMM are sensitive to occlusions.
4 .4. Audio-driven Facial Reenactment
The audio-driven reenactment extension extracts sequential motions from audio and transfers them to arbitrary individuals through PIRenderer. It produces coherent, realistic videos with varied facial movements, including lip motions and microexpressions.
- Audio-driven reenactment: PIRenderer generates coherent videos by extracting convincing sequential poses and expressions from audio and transferring them to arbitrary individuals.Sequential motions are first generated by fθ and then transformed through PIRenderer.
- Audio-driven reenactment: Compared with DAVS, the model generates audio-linked lip motions as well as head poses, eye movements, blinking, and pouting.DAVS extracts meaningful lip motions but cannot generate weakly audio-related head poses and eye movements.
- Audio-driven reenactment: Semantic coefficients let the model generate varied motions from a single audio stream and transfer them into realistic videos.The authors present this extension as evidence of PIRenderer’s potential as an efficient face renderer.
A . Additional Results of PIRenderer
Additional results show PIRenderer supports intuitive portrait editing, motion imitation, and audio-driven reenactment while producing coherent, realistic outputs. Ablation evidence indicates that continuous-frame motion descriptors improve motion accuracy and video coherence.
- Additional Results: PIRenderer supports interactive editing of facial poses and expressions, addressing modifications that commercial software may not support or may require professional skills.The supplementary results describe real-world facial image editing intended to reduce editing difficulty.
- Additional Results: PIRenderer generates coherent same-identity videos with realistic details and is more robust to driving-video occlusions than GFLA and FOMM.Cross-identity imitation also preserves the source identity while producing realistic results.
- Additional Results: Audio-driven reenactment produces accurate mouth motions, realistic eyes and head poses, and varied motions from a single input audio stream and reference image.The supplied results describe both generated videos and their input audios.
- Target Motion Descriptor: Continuous-frame 3DMM descriptors produce more accurate target motions and coherent videos than single-frame descriptors.The ablation comparison attributes the improvement to modeling temporal correlations across continuous frames.
C . Interpolation of Latent-space Z
The mapping network maps motion descriptors into a latent space where linear interpolation generates smoothly varying facial motions. Interpolation can selectively vary pose or expression while holding the other attribute fixed.
- Latent-space interpolation: The mapping network fm maps motion descriptors p to latent vectors z, and interpolated vectors z′ are formed as αfm(p1) + (1-α)fm(p2).p1 and p2 denote different motions, while α controls the interpolation.
- Latent-space interpolation: Interpolating real-world motions produces images with smooth-varying expressions and poses as α increases.Both attributes transform linearly from motion p1 toward motion p2.
- Attribute-specific interpolation: With identical expressions and different poses, interpolation preserves expression while smoothly varying pose.This demonstrates attribute-specific control in the latent space.
- Attribute-specific interpolation: With identical poses and different expressions, interpolation changes facial expression while preserving pose.The paper identifies exemplar-based portrait expression manipulation as one possible application.
D .1. Implementation Details of PIRenderer
The implementation uses a PIRenderer architecture with mapping, warping, and editing components, while sequential motion generation models temporal correlations. The renderer uses continuous-frame descriptors and normalizing-flow-based motion generation for audio-driven reenactment.
- Model Architecture: The model uses a window of k = 27 continuous frames as the center frame’s motion descriptor to address 3DMM coefficient estimation errors.The implementation sets k = 27 for all experiments.
- Training and Inference: Training pre-trains the mapping and warping networks for 200k iterations, then trains the full model end-to-end for another 200k iterations.The implementation uses ADAM with an initial learning rate of 10^-4 and batch size 20.
- Audio-driven motion generation: Normalizing flow maps latent variables and conditional information through a sequence of invertible transformations to generate motion descriptors.The flow’s conditional probability density can be explicitly computed, and training uses negative log-likelihood.
- Audio-driven motion generation: Each flow transformation combines actnorm, a linear transformation, and an affine coupling layer operating on tensors with channel and time dimensions.The affine coupling layer splits the input and predicts affine parameters with a neural network.
- Audio-driven motion generation: Audio-driven sequential motion generation models temporal correlations recurrently using previous motions and LSTM-based affine coupling networks.Separate mapping functions generate expressions and positions; the implementation sets k = 5 and τ = 6.
- Model Architecture: The PIRenderer architecture includes mapping, warping, and editing networks, with convolutional components using ADAIN or layer normalization.The mapping network transforms target motions into latent vectors, while the architecture is documented in Fig. D.11.