Source-linked AI summary
GSHeadRelight: Fast Relightability for 3D Gaussian Head Synthesis
Henglei Lv, Bailin Deng, Jianzhu Guo, Xiaoqiang Liu, Pengfei Wan, Di Zhang, Lin Gao
TL;DR
3D-aware portrait relighting must balance photorealism, real-time performance, and generalization to unseen subjects, while existing approaches face limits in supervision data and rendering. GSHeadRelight combines generative 3D Gaussians with unified neural radiance transfer, trained on in-the-wild images under a white-light assumption. The authors report state-of-the-art relighting quality and at least 12× speedup over previous methods.
Problem
3D-aware portrait relighting must balance photorealism, real-time rendering, and generalization to unseen subjects, while available light-stage supervision is limited and expensive.
Source: §1 · p. 2Method
Results
At least 12× speedup over previous methods; evaluations also report state-of-the-art 3D-aware relighting quality and better spatial consistency.
Source: §1 · p. 2, §5 · p. 10Takeaways & Limitations
The model generalizes from white-light training to colored illumination and relights without further generator evaluations for each light condition.
Source: §1 · p. 2, §5 · p. 9The spherical-harmonics radiance-transfer basis limits high-frequency reflections, and sparse below-face illumination in in-the-wild images can yield implausible relighting.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Relighting and novel view synthesis of human portraits are essential in applications such as portrait photography, virtual reality (VR), and augmented reality (AR). Despite recent progress, 3D-aware portrait relighting remains challenging due to the demands for photorealistic rendering, real-time performance, and generalization to unseen subjects. Existing works either rely on supervision from limited and expensive light stage captured data or produce suboptimal results. Moreover, many works are based on generative NeRFs, which suffer from poor 3D consistency and low real-time performance. We resort to recent progress on generative 3D Gaussians and design a lighting model based on a unified neural radiance transfer representation, which responds linearly to incident light. Using only in-the-wild images, our method achieves state-of-the-art relighting results and a significantly faster rendering speed ($\times12$) compared to previous 3D-aware portrait relighting research. Project page: http://www.geometrylearning.com/GSHeadRelight/.
1 Introduction
3D-aware portrait relighting must balance photorealism, real-time rendering, and generalization to unseen subjects, while existing approaches face limits in data access and rendering. GSHeadRelight uses generative 3D Gaussians and unified radiance transfer to address these demands.
1 Introduction
- 3D-aware portrait relighting must jointly achieve photorealism, real-time rendering, and one-shot generalization to unseen subjects.Intricate head materials and sparse real-world inputs make the lighting–material problem ambiguous.Source: §1 · p. 2
- Light-stage supervision is expensive and often inaccessible, while generative NeRF approaches can be slow and spatially inconsistent.Some casual-image approaches handle only white light or use simplified specular modeling.Source: §1 · p. 2
- GSHeadRelight uses view-conditioned unified radiance transfer per Gaussian, trained under a white-light assumption and applicable to colored illumination at test time.Because radiance responds linearly to incident light, relighting does not require further generator evaluations.Source: §1 · p. 2
- At least 12× speedup over previous methods, with ~240 FPS on a single NVIDIA H800, accompanies state-of-the-art 3D-aware relighting results.The reported speed and quality are presented as the method’s overall experimental outcome.Source: §1 · p. 2
2 Related Work
Related work spans generative 3D portrait synthesis and face relighting from studio-captured or casual images. The paper situates its approach among real-time 3D Gaussian head generators and relighting methods with varying data and rendering trade-offs.
2 Related Work
- 3D-GANs can learn fine geometry from image datasets, while EG3D’s volume rendering is too slow for native high-resolution synthesis and super-resolution can cause cross-view inconsistencies.The section notes that sparse high-quality 3D head models hinder diffusion-model training.Source: §2 · pp. 2–3
- Recent 3D Gaussian head generators support native high-resolution real-time synthesis, using either template-mesh UV maps or hierarchical anchor-constrained Gaussians.The paper builds on this recent generative 3D Gaussian progress.Source: §2 · p. 3
- Studio-captured relighting uses calibrated equipment such as light stages for reflectance supervision, while casual image collections offer greater accessibility, diversity, and scalability.The section also reviews relighting with meshes, volumetric primitives, NeRFs, and 3D Gaussians.Source: §2 · p. 3
- Casual-image approaches include style transfer, quotient-image decomposition, inverse rendering, and generative-NeRF relighting; early style-transfer methods can fail to maintain identity consistency.Diffusion-based methods also offer lighting controls such as prompts, backgrounds, and scribbles.Source: §2 · p. 3
3 Method
The method combines generative 3D Gaussian head synthesis with a unified radiance-transfer representation. It uses precomputed transfer, view-conditioned coefficients, and a white-light training assumption to support relighting under colored illumination.
3.1 Preliminaries
- Precomputed radiance transfer replaces repeated costly global-illumination computation with transfer functions applied to incident lighting at run time.The rendering equation accounts for visibility, interreflection, the BRDF, and surface orientation.Source: §3.1 · pp. 3–4
- Spherical harmonics represent lighting and transfer; earlier neural methods learn diffuse transfer and use alternatives for specular terms because transfer matrices are too large.Source: §3.1 · p. 4
- 3D Gaussian head synthesis represents points with position, scale, rotation, opacity, and color, then blends sorted overlapping points; recent generators support native-resolution real-time heads.Source: §3.1 · p. 4
3.2 Unified Neural Radiance Transfer
- Unified Neural Radiance Transfer absorbs light transport, BRDF, and cosine terms into view-dependent transfer coefficients, covering diffuse and low-frequency specular effects.Albedo is kept separate from the transfer coefficients.Source: §3.2 · p. 4
- Each Gaussian receives albedo and transfer attributes; its embedding and encoded view direction feed a decoder, and the shortened representation can be stored for splatting.Training under white light reduces the light and transfer vectors to single channels to address albedo–transfer ambiguity; linearity supports colored lighting at test time.Source: §3.2 · p. 4
4 Experiments
Experiments compare relighting quality, rendering speed, and spatial consistency with prior methods, then examine visual results and design choices. The results report fast rendering, strong comparisons, and effects of alternative lighting representations.
4.1 Implementation Details
- Implementation uses monochromatic spherical-harmonic lighting labels, foreground extraction, and a lightweight transfer decoder; training uses FFHQ for 15M images on four NVIDIA H800 GPUs.Table 1 comparisons use 512-resolution rendering except Volux-GAN at 256, on a single RTX A6000 GPU.Source: §4.1 · pp. 4–5, Table 1 · p. 5
4.2 Quantitative Comparisons
- At least 12× faster rendering than other relighting works, with similar or better generative quality than GSGAN.Table 1 compares generative quality and rendering speed; relighting speed is measured under varying illumination and view direction.Source: Table 1 · p. 5, §4.2 · p. 5
- The method is comparable in lighting quality to state-of-the-art 3D-portrait relighting methods and supports RGB relighting unavailable to implicit neural relighting methods.The authors attribute RGB relighting to the linear radiance transfer framework.Source: Table 2 · p. 5, §4.2 · p. 5
- The method achieves the highest identity similarity among compared methods, while EG3D-based methods show no disadvantage in reconstruction error despite lacking exact 3D consistency.The evaluation uses PSNR, SSIM, and identity similarity across 30 subjects and 30 views per subject.Source: Table 3 · p. 5, §4.2 · p. 5
4.3 Qualitative Comparison
- Compared with prior methods, the method achieves better relighting quality in environment-map comparisons, while alternatives show view inconsistencies, white-light limits, lighting artifacts, or lost facial details.The comparisons use the same environment maps as LumiGAN because its implementation is not publicly accessible.Source: §4.3 · p. 6, Figure 3 · p. 6
4.4 Visual Relighting Results
- White-light samples show good photorealism across different light intensities and directions, while environment-map rendering preserves head-pose consistency and produces plausible albedo and relighting.Environment maps are blurred before spherical-harmonic sampling to remove high-frequency details.Source: §4.4 · p. 6, Figure 4 · p. 7
4.5 Other Design Choices and Ablation Study
- Without explicit intrinsic supervision, RGB-light training produces faded albedo, overfits certain conditions, and fails under unseen lighting; separately modeled diffuse and specular terms are inconsistently disentangled.The RGB-light comparison trains both settings for 3M images; the diffuse/specular inconsistencies are shown in Figure 9.Source: §4.5 · p. 8, Figure 8 · p. 9, Figure 9 · p. 9
- Unified radiance transfer improves lighting manipulation accuracy, and view conditioning reduces albedo highlight residuals and yields more satisfactory relighting.The ablation compares models trained for 3M images; the complete representation achieves the best lighting manipulation accuracy and smooth albedos.Source: §4.5 · pp. 8–9, Table 4 · p. 9, Figure 10 · p. 9
5 Conclusion, Limitation, and Future Work
GSHeadRelight uses a unified radiance-transfer representation to synthesize relightable 3D-aware portraits from casual images, without light-stage data. Its results are strong, but the method is limited for high-frequency reflections and scarce below-face illumination.
5 Conclusion, Limitation, and Future Work
- The framework learns light transport from casual images under a white-light assumption, without expensive light-stage data, and avoids network evaluation for each lighting condition.Its linear response to incident lighting supports colored illumination.Source: §5 · p. 9
- GSHeadRelight achieves state-of-the-art 3D-aware relighting quality, superior real-time performance, and better spatial consistency than previous relighting methods.Its model responds linearly to incident lighting and retains light-transport consistency, supporting colored illumination.Source: §5 · pp. 9–10
- The method mainly models low-frequency diffuse effects and is not expressive enough for high-frequency reflections; below-face lighting can also seem implausible because it is scarce in in-the-wild images.In some cases, specular highlights are learned in the albedo and produce artifacts.Source: §5 · p. 10
- Future work could model high-frequency, view-dependent effects beyond spherical harmonics and adapt the method to dynamic human heads.Source: §5 · p. 10