Source-linked AI summary
FENeRF: Face Editing in Neural Radiance Fields
Jingxiang Sun, Xuan Wang, Yong Zhang, Xiaoyu Li, Qi Zhang, Yebin Liu, Jue Wang
TL;DR
Existing portrait generators trade off view consistency and local editability. FENeRF addresses this with a 3D-aware, spatially aligned representation learned from monocular image–semantic map pairs, and supports view-consistent local editing and related face-editing applications.
Problem
Portrait generation seeks high-fidelity, editable images, but 2D GANs lack view consistency while 3D-aware methods do not provide local editing.
Method
FENeRF uses decoupled shape and texture latent codes to generate a spatially aligned 3D volume, trained with paired monocular images and semantic maps.
Results
FENeRF outperforms state-of-the-art methods in several aspects and supports style mixing, style transfer, facial attribute editing, and 3D free-view editing.
Takeaways & Limitations
Jointly learning semantic and texture volumes helps generate finer 3D geometry while enabling locally editable, strictly view-consistent portraits.
Takeaways & Limitations
The generator cannot produce HD portraits, and iterative GAN inversion makes real-time free-view portrait editing an open problem.
Abstract
from arXiv · showhide
Previous portrait image generation methods roughly fall into two categories: 2D GANs and 3D-aware GANs. 2D GANs can generate high fidelity portraits but with low view consistency. 3D-aware GAN methods can maintain view consistency but their generated images are not locally editable. To overcome these limitations, we propose FENeRF, a 3D-aware generator that can produce view-consistent and locally-editable portrait images. Our method uses two decoupled latent codes to generate corresponding facial semantics and texture in a spatial aligned 3D volume with shared geometry. Benefiting from such underlying 3D representation, FENeRF can jointly render the boundary-aligned image and semantic mask and use the semantic mask to edit the 3D volume via GAN inversion. We further show such 3D representation can be learned from widely available monocular image and semantic mask pairs. Moreover, we reveal that joint learning semantics and texture helps to generate finer geometry. Our experiments demonstrate that FENeRF outperforms state-of-the-art methods in various face editing tasks.
1. Introduction
FENeRF addresses the trade-off between editable but view-inconsistent 2D portraits and view-consistent but difficult-to-edit 3D-aware portraits. It uses spatially aligned semantic and texture fields in a shared 3D representation, trained from monocular image–semantic-map pairs, to support local editing and view-consistent generation.
- 2D GANs can synthesize realistic and locally editable portraits but ignore 3D rendering, causing viewpoint-change artifacts.
- FENeRF generates strictly view-consistent portraits with interactive local editing through a shared-geometry 3D representation.The representation spatially aligns facial semantics, geometry, and texture.
- Decoupled shape and texture latent codes generate a 3D volume whose semantic and texture fields remain spatially aligned through shared geometry.A learnable 3D positional feature embedding preserves additional texture detail in synthesized portraits.
- Monocular images paired with semantic masks train the generator without requiring multi-view or 3D data.The paper states this supports data diversity and enhances the generator’s representation ability.
- Color and semantic discriminators supervise image fidelity and alignment between corresponding image content and semantic maps.
- Jointly learning semantic and texture volumes helps generate finer 3D geometry, while experiments evaluate FENeRF on CelebAMask-HQ and FFHQ.The introduction reports that FENeRF outperforms state-of-the-art methods in several aspects and supports downstream tasks.
2. Related work
Prior work advances neural implicit and 3D-aware image synthesis, including pose control and global disentanglement, but generally lacks user-interacted local editing. FENeRF combines spatially aligned semantic and texture fields with independent global styling and local facial editing while preserving view consistency.
- Neural Implicit Representations: Neural implicit representations support continuous, memory-efficient scene modeling and can be trained from posed images through differentiable volume rendering.
- Neural Implicit Representations: FENeRF builds a generative semantic field aligned with a neural radiance field to exploit spatial alignment between facial texture and semantics.
- 3D-Aware Image Synthesis: Earlier 3D-aware synthesis methods lift generation into 3D for explicit camera control, but explicit voxel or volume representations limit resolution.
- 3D-Aware Image Synthesis: Existing methods can disentangle texture and geometry globally, yet concurrent approaches generally do not support user-interacted local editing.
- 3D-Aware Image Synthesis: FENeRF supports independent global styling of texture and geometry alongside local facial attribute editing while preserving view consistency.
3. Method
FENeRF generates aligned geometry, semantics, and texture in a shared 3D volume using separate shape and texture codes, then renders images and semantic maps from arbitrary views. Its training combines adversarial supervision, pose sampling, semantic inversion, and joint rendering to preserve detail and improve geometry.
- Generator design: Two latent codes separately control geometry and semantics versus appearance, while shared intermediate features and density keep all generated fields spatially aligned.The shape code conditions density and semantic labels; the texture code conditions view-dependent color.
- Generator design: The generator uses 3D coordinates, viewing direction, and learned positional features to produce view-invariant density and semantics alongside view-dependent color.A learnable feature grid supplies additional high-frequency information for finer image details.
- Rendering: Volume rendering accumulates sampled color and semantic values along each ray to produce portrait images and semantic label probabilities from arbitrary camera poses.The shared density field is used for both color and semantic rendering, enforcing alignment in 3D space.
- Geometry interpretation: FENeRF generates smoother, more accurate geometry than π-GAN without specific regularization, including clearer separation of the 3D face from its background.The comparison highlights improvements in facial boundaries, hair, and background geometry.
- Training: Image and semantic discriminators supervise portrait fidelity and image–semantic alignment, while pose prediction adds camera-pose correction during training.The semantic discriminator receives paired synthesized images and masks; gradients from it are stopped before the color branch to preserve fine details.
- Semantic inversion: Semantic inversion evaluates 1000 real portraits through average mIoU over iterations and visualizes their recovered semantic maps from free-view viewpoints.The figure reports the inversion trajectory and shows free-view semantic results for a reference portrait.
4. Experiments
FENeRF is evaluated for image quality, semantic rendering, inversion, disentangled control, style transfer, and local facial editing. Across these tasks, it combines strong synthesis results with view-consistent, semantically guided manipulation.
- Quality evaluation: FENeRF achieves state-of-the-art image quality on CelebA-HQ and FFHQ using FID and KID.The evaluation retrains models on full datasets at 128 × 128 resolution; the authors attribute improvements to joint semantic learning and a learnable feature grid.
- Semantic rendering: Its neural semantic radiance field renders semantic maps and images from arbitrary viewpoints, with fewer extreme-pose semantic errors than SofGAN.The comparison attributes SofGAN’s inconsistencies to semantic rendering that relies on ambiguous surface construction.
- Semantic rendering: FENeRF inversion reaches average mIoU over 0.7 within 200 iterations and reconstructs facial semantics with aligned boundaries and view consistency.After 2000 iterations, the visualized examples show accurate semantic reconstruction with texture-aligned region boundaries.
- Disentangled control: Independent shape and texture codes support disentangled morphing and style mixing across geometry and appearance.Interpolation varies texture and shape separately, while style mixing combines texture from one source with shape from another.
- 3D inversion and style transfer: FENeRF preserves identity and texture more consistently during pose rotation than InterfaceGAN and E4E, with a slight drop in texture details.The latent code is tied to a 3D generative volume, so facial properties remain independent of camera pose.
- Local editing: Semantic-guided local edits, including substantial nose deformations, preserve other regions and remain strictly view-consistent.The method also transfers target texture to a source identity while maintaining consistency at poses far from the input views.
5. Limitations
FENeRF cannot currently produce HD portrait images because ray casting and volume integration are computationally expensive; real-time free-view editing also remains open due to inefficient GAN inversion.
- FENeRF cannot produce HD portrait images because ray casting and volume integration are computationally expensive.
- Real-time free-view portrait editing remains an open problem because GAN inversion uses inefficient iterative optimization.
6. Conclusion
FENeRF is presented as a locally editable 3D-aware face generator that supports semantic-guided editing and view-controlled applications. The conclusion also identifies higher-resolution portraits and 3D-aware GAN inversion as future directions, while ablations examine joint rendering and coordinate encoding.
- 6. Conclusion: Semantic maps serve as an editing interface by aligning facial semantics and texture implicitly in 3D space through shared geometry.
- 6. Conclusion: FENeRF supports style mixing, style transfer, facial attribute editing, and free-view manipulation with explicit camera control.
- 6. Conclusion: The ablations study joint semantic and image rendering, including rendered semantic maps, depth maps, images, and extracted meshes.
- 6. Conclusion: Future work targets higher-resolution synthesized free-view portraits and specific 3D-aware GAN inversion.
7. Potential Social Impact
FENeRF can turn a single real portrait into a photo-realistic avatar and drive it through semantic-map and camera-pose changes. The authors warn that resulting fake videos may pose risks for face-recognition systems and require careful deployment.
- 7. Potential Social Impact: A single real portrait can be converted into a photo-realistic avatar through GAN inversion.
- 7. Potential Social Impact: Changing semantic maps and camera poses can drive the avatar to create fake videos.
- 7. Potential Social Impact: These synthesized fake videos may fool face-recognition systems, including vivo detection, creating deployment risks.