Source-linked AI summary
Relightable Gaussian Codec Avatars
Shunsuke Saito, Gabriel Schwartz, Tomas Simon, Junxuan Li, Giljoo Nam
TL;DR
Relightable head avatars must capture complex geometry and appearance while rendering novel illumination in real time. This paper combines animatable 3D Gaussians with learnable radiance transfer and explicit eye models, achieving high-fidelity all-frequency relighting that outperforms evaluated real-time alternatives. The approach still depends on preprocessing and precise illumination information for some settings.
Problem
Existing geometry and appearance representations struggle with intricate head structures, diverse materials, high-frequency reflections, and real-time relighting under continuous illumination.
Method
The method combines animatable 3D Gaussian geometry, learnable radiance transfer using spherical harmonics and spherical Gaussians, and explicit relightable eye models.
Results
The proposed combination outperforms evaluated real-time geometry and appearance combinations while enabling high-fidelity relighting of hair, skin, and eyes under all-frequency illumination.
Takeaways & Limitations
The approach supports real-time relighting of animatable head avatars with detailed geometry, all-frequency reflections, and explicit gaze control.
Takeaways & Limitations
The approach requires a coarse mesh and gaze tracking during preprocessing, and in-the-wild use remains challenging because precise illumination is unavailable.
Abstract
from arXiv · showhide
The fidelity of relighting is bounded by both geometry and appearance representations. For geometry, both mesh and volumetric approaches have difficulty modeling intricate structures like 3D hair geometry. For appearance, existing relighting models are limited in fidelity and often too slow to render in real-time with high-resolution continuous environments. In this work, we present Relightable Gaussian Codec Avatars, a method to build high-fidelity relightable head avatars that can be animated to generate novel expressions. Our geometry model based on 3D Gaussians can capture 3D-consistent sub-millimeter details such as hair strands and pores on dynamic face sequences. To support diverse materials of human heads such as the eyes, skin, and hair in a unified manner, we present a novel relightable appearance model based on learnable radiance transfer. Together with global illumination-aware spherical harmonics for the diffuse components, we achieve real-time relighting with all-frequency reflections using spherical Gaussians. This appearance model can be efficiently relit under both point light and continuous illumination. We further improve the fidelity of eye reflections and enable explicit gaze control by introducing relightable explicit eye models. Our method outperforms existing approaches without compromising real-time performance. We also demonstrate real-time relighting of avatars on a tethered consumer VR headset, showcasing the efficiency and fidelity of our avatars.
1. Introduction
Real-time relighting of animatable human heads remains difficult because facial materials and moving geometry are complex, while existing approaches struggle to combine fidelity, diverse materials, and efficient rendering. The paper proposes Gaussian geometry, learnable radiance transfer, and explicit eye modeling to address these challenges.
- Motivation: Real-time relighting must model detailed facial appearance coherently with illumination while supporting applications such as games and telecommunication.The challenge is especially acute because visual perception is highly sensitive to facial appearance.
- Motivation: Skin, hair, and eyes have distinct scattering, reflection, and geometric properties that are difficult to represent with one real-time material model.Skin involves micro-geometry and subsurface scattering, hair has translucent fiber scattering, and eyes contain reflective layered membranes.
- Prior limitations: Physically based methods generalize to novel illumination but make dynamic capture, non-skin modeling, and accurate geometry and material acquisition laborious.These methods require exhaustive measurements and substantial manual cleanup.
- Contributions: The proposed framework combines drivable 3D Gaussian avatars, learnable radiance transfer, and explicit eye models for real-time relighting.The design targets intricate geometric details, global light transport, all-frequency reflections, and disentangled gaze control.
- Contributions: Dynamic spherical harmonics model diffuse radiance transfer, while spherical Gaussians represent specular effects with view-dependent visibility.The visibility term approximates occlusion, Fresnel, and geometric attenuation without explicitly estimating each component.
- Results: The combination of 3D Gaussian geometry and the proposed appearance model outperforms other evaluated combinations while remaining real-time renderable.The comparison includes geometry and appearance models from this work and existing real-time baselines.
2. Related Work
Prior work spans facial modeling, reflectance capture, neural relighting, and precomputed radiance transfer, but important gaps remain in thin-structure modeling, novel animation, and expressive real-time illumination.
- Face Modeling: Image-based facial reconstruction enables precise acquisition, but traditional approaches primarily focus on facial skin and require tailored solutions for other head components.The related-work discussion identifies broader head modeling as an ongoing need.
- Face Modeling: Existing real-time shape representations struggle to model extremely thin structures such as hair strands.The paper extends 3D Gaussian splatting to animatable and relightable facial avatars to address this limitation.
- Facial Reflectance Capture: Reflectance capture based on one-light-at-a-time measurements uses light-transport linearity to relight faces under novel illumination.Follow-up work extended this paradigm toward dynamic relighting and faster acquisition.
- Neural Relighting: Neural relighting learns appearance directly from light-stage captures, but many methods do not support novel animations or temporally coherent dynamic relighting.Single-image portrait relighting also cannot provide novel-view synthesis and temporally coherent dynamic relighting.
- Neural Relighting: Environment-map relighting methods often rely on teacher-student distillation, while their lighting input can limit expressiveness for all-frequency relighting.The paper instead integrates target illumination at the neural decoder output, inspired by precomputed radiance transfer.
- Precomputed Radiance Transfer: Precomputed radiance transfer accelerates global illumination by combining precomputed intrinsic scene factors with extrinsic illumination at runtime.This separates reusable geometry and reflectance-dependent transport from changing illumination.
3. Method
The method combines animatable 3D Gaussian geometry with learnable radiance-transfer appearance and explicit eye modeling. It decodes these representations from expression and gaze inputs, relights them under illumination, and renders the result with splatting-based supervision.
- Geometry: 3D Gaussian Avatars: The avatar geometry uses anisotropic 3D Gaussians, parameterized on a shared UV map and decoded to represent thin structures and dynamic expressions.A CVAE provides expression latents, while decoders predict Gaussian transformations, opacity, and coarse mesh vertices; Gaussian positions use coarse-mesh guidance.
- Geometry: 3D Gaussian Avatars: Each Gaussian is projected into a 2D splat and composited with cumulative volumetric rendering to produce image colors.The projected covariance is computed from the viewing transformation and projective Jacobian, and ordered splats are combined using opacity-weighted accumulation.
- Appearance: Learnable Radiance Transfer: The appearance model decomposes color into diffuse and specular components to represent subsurface scattering, shadows, and view-dependent reflections.Diffuse transfer uses spherical harmonics with higher-frequency intrinsic coefficients for shadows, while specular transfer uses spherical Gaussians and learnable visibility.
- Appearance: Learnable Radiance Transfer: View-dependent normals allow one Gaussian representation to model both surface reflections and fiber-like hair reflections without predefining the reflection type.The normal may remain fixed for surfaces but rotate along a fiber tangent as the view changes.
- Relightable Explicit Eye Model: An explicit eye model supports high-fidelity eye relighting, while training combines rendered-image reconstruction losses with coarse-geometry supervision.The reconstruction objective includes L1, D-SSIM, and L2 losses on the decoded coarse geometry.
4. Experiments
Experiments evaluate held-out expressions and lights, compare geometry and appearance representations, and visualize intrinsic decomposition. The results show detailed geometry, realistic eye glints, and high-frequency relighting supported by the proposed representation.
- Evaluation Protocol: The evaluation uses unseen conversational and disgust expressions, excludes 10 frontally biased light patterns, and reports PSNR, SSIM, and LPIPS.The protocol includes approximately 9,000 conversational frames, 100 disgust frames, and 1,800 excluded frames.
- Qualitative Results: The avatar generalizes to novel views, expressions, point lights, and high-resolution environment maps while supporting real-time driving from a headset.The qualitative results include mirror-like eye reflections that preserve high-frequency environmental details.
- Geometric Representation: 3D Gaussian geometry better models skin details and hair strands than voxel-based MVP, while the explicit eye model improves eye glints.The comparison changes only geometric representation while keeping the appearance model fixed.
- Intrinsics Decomposition: The full render decomposes into diffuse and specular components, with diffuse shading formed by learned albedo and SH-based radiance transfer.Specular-lobe directions are computed from per-Gaussian normals.
- Appearance Representation: Spherical Gaussians provide high-frequency reflections, avoiding the band limitation of spherical harmonics for specularity.Compared with a linear model and EyeNeRF, the method avoids blurry results and flickering artifacts associated with limited or overfit appearance representations.
5. Conclusion
The paper concludes that combining 3D Gaussian geometry with spherical-harmonic and spherical-Gaussian radiance transfer enables real-time, high-fidelity relighting of hair, skin, and eyes. It also identifies preprocessing, illumination knowledge, and multi-avatar scaling as practical boundaries.
- Conclusion: The proposed representation supports real-time rendering of high-fidelity relighting for hair, skin, and eyes under all-frequency illumination.The appearance basis combines spherical harmonics and spherical Gaussians.
- Conclusion: 3D Gaussian Splatting is critical for strand-accurate hair reconstruction and relighting.The conclusion reports significant qualitative and quantitative improvements over existing real-time-renderable geometry and appearance models.
- Limitations and Future Work: The method requires a coarse mesh and gaze tracking during preprocessing, which may be sensitive to tracking failures.The authors identify end-to-end learning and topology-consistent tracking as future directions.
- Limitations and Future Work: In-the-wild inputs remain challenging because precisely known illumination information is unavailable.Rendering many Gaussian avatars is also difficult because relighting is performed per Gaussian and scales linearly with avatar count.
A. Network Architecture
The head decoder separates view-independent and view-dependent processing while injecting gaze features only into eye regions. Expression and gaze latents are transformed into spatial feature tensors for decoding.
- View-Independent Decoder: An expression latent z ∈ R256 is projected through a leaky-ReLU layer and reshaped into a 256 × 8 × 8 feature tensor.This forms the expression features used by the head decoder.
- Gaze Conditioning: Each eye’s gaze direction is projected through a leaky-ReLU layer and reshaped into a 16 × 2 × 2 feature tensor.The same gaze processing is applied separately to each eye.
- Gaze Conditioning: Gaze features are concatenated only at eyeball locations in UV space, with zeros elsewhere.This spatially localizes gaze conditioning to the eye regions.
B. Discussion: Appearance Representation
The appearance discussion derives a spherical-Gaussian approximation for specular reflection from the rendering equation. A learned view-dependent factor then combines with incoming illumination and a spherical-Gaussian lobe to model reflections efficiently beyond a simple microfacet formulation.
- Rendering Equation: The rendering equation integrates incoming radiance, visibility, BRDF, and cosine-weighted illumination over the sphere of directions.The prose defines ω_i and ω_o as incoming and outgoing directions, L as incoming intensity, V as visibility, ρ as BRDF, and n as surface normal.
- Specular Approximation: The specular BRDF is expressed with Fresnel, geometric attenuation, and a normal-distribution function represented by a spherical Gaussian.The spherical-Gaussian NDF is centered on the reflection vector q with bandwidth σ.
- Specular Approximation: The resulting specular term integrates illumination against a spherical-Gaussian lobe weighted by visibility, geometric attenuation, and cosine shading.This follows from substituting the spherical-Gaussian approximation into the rendering equation.
- Efficient Formulation: When σ ≪ 1, the integral concentrates near the reflection direction, allowing view-dependent terms to be combined into v_k while retaining incoming radiance and the NDF.The resulting expression separates the learned view-dependent factor from the illumination integral.
- Efficient Formulation: A neural network parameterizes v_k(ω_o), enabling end-to-end optimization and unified modeling of diffuse and highly reflective regions.The authors report that the formulation is fast to compute and stable to optimize, while extending beyond general microfacet or single-bounce reflection.
C. Ablation Study
The ablations validate key appearance, geometry, and capture-design choices, while also identifying components with limited or diminishing benefits. The model remains effective with reduced tracking and moderate camera counts, though quality degrades under more severe reductions.
- Higher-order Monochrome SH: Higher-order monochrome SH coefficients improve shadow precision and overall reconstruction accuracy for high-frequency environments.The coefficients extend to 8th order; removing the 4th–8th orders produces less precise shadows.
- View-dependent Normal: The view-conditioned surface normal does not produce clear qualitative or quantitative improvements in the evaluated comparisons.The authors hypothesize that view-conditioned visibility compensates for errors from view-independent normals in cylindrical regions.
- Spherical Gaussian Representation: The angle-based spherical Gaussian formulation recovers eye glints more accurately than the conventional cosine-based formulation.This comparison is shown against a held-out frame in the spherical Gaussian ablation.
- Rigid Tracking: Using only rigid head-pose tracking and a generic template still achieves good reconstruction, but poorer registration can blur extreme expressions and remove eye glints.The dependency on accurate non-rigid surface tracking can be optionally removed at a slight quality cost.
- Number of Cameras: At least 32 cameras yield good novel-view results, whereas 8 cameras noticeably degrade quality and 16 cameras introduce artifacts, especially around the eyes.Using more than 32 cameras provides diminishing returns.
- Number of Lighting Conditions: The lighting-condition ablation evaluates held-out illuminations across training sets ranging from 10+1 to 577+1 conditions.The study varies partial and uniform illuminations, but temporal multiplexing changes the number of training frames and prevents holding out physical lights independently.
D. Performance
The performance section reports the computational trade-off between Gaussian count and rendering speed, alongside the lighting-condition configurations used in the evaluation.
- Gaussian Count: 1024×1024 = 1 Mi Gaussians require 12.84 ms for splatting, while 512×512 = 256 Ki Gaussians require 6.40 ms on an NVIDIA A100.The smaller model is used for the VR demo to improve framerate, while the larger model improves quality at the cost of slower decoding and rendering.
- Lighting Conditions: The lighting ablation evaluates 10+1, 30+1, 120+1, 360+1, and 577+1 light-condition configurations against a ground-truth image.The results are shown on held-out illuminations for a training frame and camera.
- Convergence: MVP-based models require 400 K iterations for convergence, described as twice as many iterations as the comparison model.
E. Ethical Concerns
The authors limit the model to a small set of consenting subjects captured with dense multiview systems, with personalized expression spaces restricting intended use to self-driven avatars.
- Scope and Consent: The model is evaluated on only a few consenting subjects captured in a dense multiview capture system.The expression latent space is personalized for each individual, limiting use to driving one’s own avatar with consent.