Source-linked AI summary
Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars
Jingxiang Sun, Xuan Wang, Lizhen Wang, Xiaoyu Li, Yong Zhang, Hongwen Zhang, Yebin Liu
TL;DR
The paper addresses accurate, identity-preserving animation of high-fidelity facial avatars learned from unstructured 2D images. It introduces Generative Texture-Rasterized Triplanes with mouth-interior modeling, achieving strong 3D-consistent synthesis and animation while supporting downstream avatar applications.
Problem
Animatable portrait generators must control pose, expressions, and gaze finely while modeling accurate deformation and preserving identity from unstructured 2D images.
Method
The method learns Generative Neural Textures on parametric meshes, rasterizes them into tri-plane features for volumetric rendering, and separately models mouth interiors.
Results
The method achieves the best performance on all reported metrics, including pose consistency of 0.019 for yaw and pitch, while matching unconditional 3D GAN FID quality at 4.7.
Takeaways & Limitations
The learned animatable 3D representation serves as a 3D prior for applications including one-shot facial avatars and animatable out-of-domain avatar generation.
Takeaways & Limitations
The model struggles with full-consistency extrapolation for challenging expressions such as one-side mouth up, frown, and sticking the tongue out.
Abstract
from arXiv · showhide
3D-aware generative adversarial networks (GANs) synthesize high-fidelity and multi-view-consistent facial images using only collections of single-view 2D imagery. Towards fine-grained control over facial attributes, recent efforts incorporate 3D Morphable Face Model (3DMM) to describe deformation in generative radiance fields either explicitly or implicitly. Explicit methods provide fine-grained expression control but cannot handle topological changes caused by hair and accessories, while implicit ones can model varied topologies but have limited generalization caused by the unconstrained deformation fields. We propose a novel 3D GAN framework for unsupervised learning of generative, high-quality and 3D-consistent facial avatars from unstructured 2D images. To achieve both deformation accuracy and topological flexibility, we propose a 3D representation called Generative Texture-Rasterized Tri-planes. The proposed representation learns Generative Neural Textures on top of parametric mesh templates and then projects them into three orthogonal-viewed feature planes through rasterization, forming a tri-plane feature representation for volume rendering. In this way, we combine both fine-grained expression control of mesh-guided explicit deformation and the flexibility of implicit volumetric representation. We further propose specific modules for modeling mouth interior which is not taken into account by 3DMM. Our method demonstrates state-of-the-art 3D-aware synthesis quality and animation ability through extensive experiments. Furthermore, serving as 3D prior, our animatable 3D representation boosts multiple applications including one-shot facial avatars and 3D-aware stylization.
1. Introduction
Next3D targets accurate, controllable facial animation from unstructured 2D images by combining mesh-guided deformation with volumetric flexibility. It also models mouth interiors and supports downstream avatar applications.
- Animatable portrait generators must control head pose, expressions, and gaze while preserving deformation accuracy and identity from unstructured 2D images.
- Next3D proposes an unsupervised 3D GAN framework for high-quality, 3D-consistent facial avatars from unstructured 2D images.
- The framework models mouth interiors omitted by 3DMM and adds a deformation-aware discriminator to encourage alignment with expected deformation.
- The method provides fine-grained animation including expressions, eye blinks, gaze direction, and full head poses, while supporting one-shot avatars and 3D stylization.
- Generative Texture-Rasterized Triplanes combine mesh-guided expression control with implicit volumetric flexibility by rasterizing neural textures into three orthogonal feature planes.
2. Related Work
Prior work spans 3D-aware generation, editable facial animation, and explicit or volumetric scene representations. Next3D combines explicit mesh guidance with continuous volumetric features to retain deformation accuracy while allowing topology changes.
- Modern 3D-aware generators use efficient neural scene representations such as implicit networks, sparse voxel grids, and multiple planes for high-resolution synthesis.
- Editable facial generators incorporate semantic radiance fields, GAN inversion, or 3D priors, but some cannot provide continuous and stable video editing.
- Next3D adapts explicit surface deformation into a continuous volumetric representation, enabling accurate animation together with topology changes.
- 3DMM-based methods model facial deformation accurately but struggle with non-facial areas such as hair, teeth, eyes, and body.
- Volumetric representations store occupancies, radiance, or colors throughout space rather than only on an object surface.
3. Approach
The approach separates dynamic facial parts from static scene components, rasterizes generative neural textures into tri-planes, completes mouth interiors, and renders blended features volumetrically. Training combines adversarial, deformation-aware, density, and image-based objectives.
- Dynamic and static components are modeled with separate tri-plane branches, while the mouth synthesis module completes interior features absent from 3DMM.
- Generative Texture-Rasterized Tri-planes rasterize StyleGAN-synthesized neural textures from a deformable template mesh into three orthogonal feature planes.
- The representation converts surface deformation into a continuous volume, retaining mesh guidance while supporting topological flexibility such as glasses.
- The teeth module completes missing mouth features from cropped tri-plane regions, and neural blending reduces mouth-boundary texture flickering.
- The teeth synthesis module improves both animation accuracy and synthesis quality, while training uses non-saturating GAN, R1, and density regularization objectives.
- The blended tri-planes are sampled, decoded into volume density and features, volume-rendered into feature images, and super-resolved from 64 × 64 to 512 × 512 RGB images.
4. Experiments
Experiments compare Next3D with animatable 2D and 3D baselines, evaluate its image quality, animation fidelity, identity consistency, and design choices, and demonstrate avatar and stylization applications.
- Comparisons: Next3D outperforms DiscoFaceGAN, 3DFaceShop, and AniFaceGAN in qualitative synthesis quality and animation accuracy.It preserves detailed driver expressions, including mouth interior, eye blinks, eye movements, and in-plane head rotations.
- Comparisons: Next3D models detailed eyelid and lip deformations while preserving unchanged glasses, demonstrating topology-aware dynamic geometry.The representation combines explicit mesh guidance with continuous volumetric flexibility.
- Ablation study: Static tri-planes improve identity consistency, the teeth module improves mouth synthesis, and synthetic-rendering discrimination improves detailed-expression reconstruction.Removing static tri-planes entangles identity with expression, while the full design advances animation and synthesis quality.
- Comparisons: Next3D achieves the best performance on FID, AED, APD, and identity consistency among the evaluated methods.Its FID reaches the level of unconditional 3D GANs, and pose consistency remains 0.019 when evaluation considers only yaw and pitch.
- Applications: The learned representation supports one-shot portrait animation and text-driven 3D-aware stylization while preserving consistent animation properties.PTI fits avatars to real portraits, and CLIP-guided adaptation produces stylized portraits from textual prompts.
5. Limitations and future work
The method extrapolates reasonably to some rare expressions but remains inconsistent on several challenging facial movements.
- Limitations and future work: Next3D struggles to model one-sided mouth raising, frowning, and tongue protrusion with full consistency.The authors suggest richer-expression videos and a more powerful face model as future directions.
6. Conclusion
Next3D provides an animatable 3D representation for learning high-quality, 3D-consistent facial avatars from unstructured 2D images.
- Conclusion: The learned representation is presented as a strong 3D prior for 3D-aware one-shot facial avatars and animatable out-of-domain avatar generation.These applications are identified as downstream uses of the learned representation.
A. Additional experiments
Additional experiments study deformation-aware discriminator designs and compare training variants using synthetic renderings and FLAME-parameter conditioning.
- Additional experiments: The deformation-aware discriminator is evaluated by comparing FID trends from a shared checkpoint across conditioning designs.The baseline is trained without synthetic renderings or FLAME-parameter conditioning before the alternatives are tested.
- Additional experiments: Training convergence is examined under different discriminator designs.The convergence comparison is summarized in Figure 9.
A.2. Training strategy of 3D-aware stylization
The stylization ablation compares freezing all toRGB layers with optimizing the final toRGB layer in each synthesis network. Optimizing the final layers improves synthesis quality.
- Training strategy: Figure 10 presents the ablation study comparing the two layer-freezing strategies for 3D-aware stylization.The comparison evaluates the default strategy against the alternative final-layer optimization strategy.
- Training strategy: Freezing all toRGB layers, the default StyleGAN-NADA strategy, degraded image quality and introduced dissymmetry in 3D-aware stylization.This strategy works in 2D space but was unsuitable for the reported 3D setting.
- Training strategy: The alternative strategy optimizes the last toRGB layer for each of the three synthesis networks.The optimized layers belong to the neural texture generator Guv, static tri-plane generator Gstatic, and teeth completion module Gteeth.
- Training strategy: The three synthesis networks are Guv, Gstatic, and Gteeth, whose final toRGB layers are included in optimization.This specifies the trainable exception to the otherwise frozen synthesis-network layers.
B.1. Data preprocessing
Data preprocessing uses FLAME and DECA for facial deformation, while eye poses are optimized from eye landmarks because suitable eye-pose extraction is unavailable. A dedicated teeth module completes mouth-interior features.
- Facial and eye preprocessing: FLAME drives facial deformation, and DECA extracts the FLAME parameters used to obtain template meshes.The preprocessing pipeline therefore supplies parametric facial geometry for the model.
- Facial and eye preprocessing: An eye landmark detector extracts five landmarks around each eye, which correspond to five selected template-mesh vertices.The eye-pose variables optimized from these correspondences are yaw and pitch.
- Mouth-interior preprocessing: The teeth completion module Gteeth encodes concatenated tri-plane teeth textures into multiscale feature maps before generating a 64 × 64 × 768 feature map.Its architecture uses residual blocks between the encoder and the right-side generator.
C. Experiment details
Experiment details cover one-shot-avatar preprocessing, figure-based eye-landmark detection, and the architectures of the teeth module and stylization setup. Real portraits are aligned to the training data before inversion and generator tuning.
- One-shot avatars: For one-shot facial avatars, an off-the-shelf face detector extracts camera poses and crops in-the-wild portraits to match the training-set format.This preprocessing prepares real portraits for the inversion-based avatar pipeline.
- Eye preprocessing: Eye-pose preprocessing uses five landmarks around the eyes, as illustrated in Figure 11.The landmarks support optimization of eye yaw and pitch on the template mesh.
- Inversion and stylization: After extracting FLAME parameters and template meshes with DECA, the pipeline optimizes the latent code for 450 iterations and fine-tunes generator weights for 500 iterations.The stylization procedure then optimizes partial generator weights while fixing the others.
D. Additional visual results
Additional visual results show selected expressions and poses, qualitative baseline comparisons, animated avatars, and applications to real-portrait one-shot avatars and 3D-aware stylization. The examples emphasize controllability, shape quality, and downstream use.
- Generated expressions and poses: Selected examples demonstrate image quality, expression controllability, gaze animation, and output diversity across expressions and poses.These examples are presented in Figure 13.
- Qualitative comparison: Figure 14 provides a qualitative comparison of facial animation against baselines.The comparison is part of the supplementary visual results.
- Animated avatars: Animated virtual avatars exhibit high-quality shapes, with eyelid motions reflected on the extracted meshes.The examples are provided in Figure 15.
- Animated avatars: Rotated eyeballs in the template mesh provide an explicit gaze-direction signal and help decouple gaze from pose during inference.The passage connects this design to alleviating the hollow-face illusion.
- Downstream applications: Figure 16 shows one-shot avatars for real portraits and results from 3D-aware stylization.The supplementary material also directs readers to accompanying dynamic results.