PaperlayerView exact revision on arXiv

Source-linked AI summary

ARS-Avatar: Animatable and Relightable Surfel Avatars with Learnable Ambient Occlusion

Jiateng Liu, Hao Gao, Junxin Sun, Mengqi Liu, Jiu-Cheng Xie, Jucheng Song, Feng Xu

arXiv:2609.27600v1cs.GR

TL;DR

Creating animatable, relightable avatars from multi-view images under unknown illumination is challenging because deformation, materials, and visibility are coupled. ARS-Avatar combines deformation priors, surfel-based deferred shading, and differentiable screen-space occlusion. Experiments report state-of-the-art reconstruction quality and photorealistic animation and relighting.

  • Problem

    Reconstructing avatars from multi-view images under unknown illumination is ill-posed because geometry, materials, and illumination are tightly coupled, while pose-only inputs underrepresent clothed-human deformation.

    Source: §1 · pp. 1–2

  • Method

    ARS-Avatar uses template-derived deformation priors to estimate Gaussian-surfel attributes and combines deferred BRDF shading with differentiable SSAO radius optimization.

    Source: §1 · p. 2

  • Results

    Experiments on two datasets demonstrate state-of-the-art reconstruction quality and photorealistic animation and relighting.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The conclusion reports high appearance and geometry consistency with faster rendering than implicit methods, alongside realistic animation and relighting.

    Source: §5 · p. 8

    Deformation-prior extraction relies on predefined priors that may not capture diverse information across clothed bodies.

    Source: §5 · p. 8

Abstract

from arXiv · show

Creating animatable and relightable human avatars from multi-view images remains challenging, as pose-dependent deformation, materials, and light visibility are intrinsically coupled in images. In this paper, we present ARS-Avatar, a novel method using surfel representation for high-quality, animatable, and relightable human avatars from multi-view images captured under unknown illumination. We first extract deformation priors from the template mesh and leverage as additional details beyond driving poses to facilitate faithful estimation of surfel attributes and reconstruction of animatable avatar. To support relighting, the deferred shading is employed to estimate BRDF materials. We further introduce a differentiable screen-space ambient occlusion formulation that enables gradient-based optimization of body-part specific occlusion radii through finite differences, providing an efficient approximation of light visibility that can be jointly optimized with the avatar. Extensive experiments demonstrate that ARS-Avatar achieves high-fidelity appearance reconstruction and physically-based material estimation, while enabling realistic animation and relighting under novel poses and illuminations.

1. Introduction

Reconstructing animatable, relightable avatars from multi-view images under unknown illumination is ill-posed because geometry, materials, and lighting are coupled. ARS-Avatar combines deformation priors, surfels, deferred shading, and optimizable ambient occlusion; experiments report state-of-the-art reconstruction quality and photorealistic animation and relighting.

1. Introduction

  • Multi-view reconstruction under unknown illumination is ill-posed because geometry, material properties, and scene illumination are tightly coupled.Traditional methods rely on high-quality 3D scans and calibrated illumination.Source: §1 · p. 1
  • NeRF-based methods achieve impressive quality, but expensive volume rendering motivates efficient explicit representations.Recent approaches combine primitives with templates and physically based rendering.Source: §1 · p. 1
  • Position maps or driving poses alone do not fully represent the rich pose-dependent deformation of clothed humans.The paper notes that this limitation constrains animation and relighting fidelity, alongside the difficulty of extending visibility estimation to discrete Gaussian surfels.Source: §1 · p. 2
  • ARS-Avatar uses Gaussian surfels, template-derived deformation priors, deferred shading, and differentiable screen-space ambient occlusion to build animatable, relightable avatars.The priors are deformation displacement and deviation from the parametric body model; occlusion radii are optimized with finite differences.Source: §1 · p. 2
  • Experiments on two datasets show state-of-the-art reconstruction quality and photorealistic animation and relighting.The conclusion also reports high fidelity and consistency in appearance and geometry, with faster rendering than implicit methods.Source: §1 · p. 2

2. Related Work

Prior work progresses from template-based and neural avatar representations to physically based relighting and explicit visibility estimation. ARS-Avatar addresses pose-dependent deformation and differentiable occlusion within a Gaussian-surfel representation.

2.1. Animatable Human Avatar Modeling

  • Driving-pose conditioning alone does not capture rich deformation in clothed humans, motivating deformation priors for more faithful pose-dependent reconstruction.The cited progression runs from explicit meshes and template body models through learnable deformation fields and NeRF to efficient explicit 3DGS avatars.Source: §2.1 · p. 2

2.2. Relightable Human Avatar Modeling

  • Relightable-avatar research combines animation and physically based material and illumination models; this work uses surface-aligned Gaussian Surfels as its representation.Earlier approaches include neural relighting of single images, physically based rendering in NeRF, explicit light transport, and Gaussian avatars.Source: §2.2 · p. 2

2.3. Visibility and Occlusion Estimation

  • Existing visibility methods often detach visibility from avatar optimization, while this method makes SSAO radius differentiable through finite differences.Prior approaches include mesh ray tracing, NeRF visibility, ambient-occlusion representations, and screen-space ambient occlusion.Source: §2.3 · p. 2

3. Method

ARS-Avatar estimates pose-dependent surfel attributes from template-derived maps, then uses deferred shading and differentiable screen-space ambient occlusion for relighting. Its training combines image, geometry, deformation, lighting, and regularization losses.

3.1. Overview

  • The pipeline estimates surfel attributes from pose and deformation-prior maps, estimates BRDF materials with deferred shading, and adds jointly optimized SSAO.The overview describes the sequence from a posed canonical template through surfel estimation and material estimation to occlusion optimization.Source: Figure 2 · p. 3, §3.1 · p. 3

3.2. Surfel Estimation with Deformation Priors

  • Template-derived priors encode deformation displacement and distance to the nearest SMPL-X vertex, and their learned feature maps guide surfel-attribute estimation alongside position maps.The canonical surfels are also deformed into observation space using linear blend skinning.Source: §3.2 · p. 3

3.3. Material Estimation with Deferred Shading

  • Deferred shading estimates BRDF materials using albedo and roughness, with a learnable HDR cube map representing incident illumination.The shaded color combines diffuse and specular components, then ambient occlusion modulates the result.Source: §3.3 · p. 4

3.4. Differentiable Screen-space Ambient Occlusion

  • SSAO radius optimization adjusts nine body-part radii while holding depth and normal inputs fixed, estimating the radius gradient with finite differences.Surfel radii blend neighboring joint radii by linear-blend-skinning weights; the resulting occlusion modulates the physically based shading color.Source: §3.4 · p. 4

3.5. Loss Function and Optimization

  • Training combines L1, perceptual, geometry, normal-deformation, regularization, total-variation, and lighting losses.The geometry loss enforces rendered normal and depth consistency, while the normal-deformation term regularizes neighboring surfel normals.Source: §3.5 · p. 4

4. Experiments

Experiments compare avatar reconstruction and physically based relighting against prior methods, then ablate deformation priors, screen-space ambient occlusion, and surfel representation. ARS-Avatar reports superior radiance metrics and more realistic novel-lighting effects, with ablations supporting the role of its components.

4. Experiments

  • Experiments use AvatarReX and ActorsHQ, with four AvatarReX characters and two ActorsHQ actors selected for training.Evaluation metrics are PSNR, SSIM, LPIPS, and FID.Source: §4 · pp. 4–5

4.1. Comparison

  • ARS-Avatar achieves superior radiance metrics and high-fidelity results under novel poses, especially for loose garments and articulated parts.The comparison includes 3DGS-Avatar [46], WebAvatar, and AnimatableGaussians [33].Source: Table 1 · p. 5, §4.1 · p. 6, Figure 4 · p. 6
  • ARS-Avatar produces more realistic illumination and occlusion under novel lighting than the compared physically based avatar methods.The comparison includes MeshAvatar and Relighting4D; its screen-space ambient occlusion is jointly optimized with the avatar.Source: §4.1 · p. 6, Table 2 · p. 6, Figure 5 · p. 7

4.2. Ablation Study

  • Deformation priors improve reconstruction by adding spatial guidance for pose-dependent surfel attributes, preserving fine details in regions such as limbs.The reported effect is strongest in regions with large-range movement.Source: §4.2 · pp. 6–7, Figure 7 · p. 8
  • Jointly optimizing ambient occlusion improves results over fixed SSAO while retaining efficient screen-space computation.The qualitative comparison includes the full method, a variant without optimizable radius offsets, and other occlusion approaches.Source: §4.2 · pp. 7–8, Figure 6 · p. 8
  • Gaussian Surfels produce more coherent depth and normal maps with sharper geometric boundaries than conventional Gaussians, benefiting material estimation and visibility modeling.The comparison converts depth to normals for clearer visualization.Source: §4.2 · p. 8, Figure 8 · p. 8

5. Conclusion

ARS-Avatar uses surfel primitives and differentiable ambient occlusion for avatar rendering. The conclusion reports fidelity, consistency, and speed benefits, while noting that predefined deformation priors do not capture diverse information across clothed bodies.

5. Conclusion

  • Surfel primitives provide high-fidelity, appearance-and-geometry-consistent avatars with faster rendering than implicit methods.Source: §5 · p. 8
  • Differentiable ambient occlusion estimates shadows within 3DGS optimization without extra components while maintaining efficiency.Source: §5 · p. 8
  • Predefined deformation priors do not capture diverse information across clothed human bodies.Source: §5 · p. 8

6. Implementation Details

The implementation uses PyTorch and a single RTX 3090 GPU, with custom CUDA extensions for rasterization, cubemap filtering, and differentiable ambient occlusion.

6. Implementation Details

  • All models are implemented in PyTorch and trained on a single NVIDIA RTX 3090 GPU with 24 GB of memory.Source: §6 · p. 12
  • The rasteriser, cubemap filtering, and differentiable ambient occlusion formulation use custom CUDA extensions.Source: §6 · p. 12
  • Split-sum pre-filtering uses the public nvdiffrec release's cosine and GGX kernels, and pose PCA is applied to position and deformation-prior maps.Source: §6 · p. 12

7. Detailed Deformation Feature Maps

Deformation feature maps encode two template-derived distances, retaining displacement direction and using only the magnitude of the nearest-vertex distance.

7. Detailed Deformation Feature Maps

  • The two deformation distances are displacement before versus after deformation, d, and distance to the nearest SMPL-X vertex, s.Source: §7 · p. 12
  • The final encoding includes the magnitude and orientation of d, but only the magnitude of s because s measures LBS deformation-weight error.Source: §7 · p. 12
  • The feature vector concatenates ||d||, d's direction, and ||s||.Source: §7 · p. 12

8. Detailed Splatting Formulation

The splatting formulation renders appearance, geometry, material, and ambient-occlusion-related maps; albedo and roughness are splatted without background influence.

8. Detailed Splatting Formulation

  • Splat renders color, normal, and depth maps from surfels and extends the outputs to albedo, roughness, position, radii, sample importance, and SSAO direction maps.Source: §8 · p. 12
  • The rendered outputs include perturbed radius maps R+ and R− alongside the radius map R.Source: §8 · p. 12
  • The formulation splats albedo and roughness without background influence to avoid background-noise training failure during deferred shading.Source: §8 · p. 12

9. Detailed Physically-Based Shading

The shading formulation separates diffuse and specular BRDF contributions, computes their colors, and combines them into a shaded color before visibility is applied.

9. Detailed Physically-Based Shading

  • The specular color uses a SplitSum approximation to separate lighting and BRDF integrals, with the specular BRDF formed from D, F, and G.D, F, and G denote the normal distribution, Fresnel, and geometry terms.Source: §9 · p. 12
  • The shaded color is the sum of diffuse and specular colors, combined before accounting for visibility.The BRDF is likewise decomposed into diffuse and specular components.Source: §9 · p. 12
  • With metallic set to zero, the diffuse BRDF simplifies to albedo divided by π.The diffuse color integrates incident illumination weighted by the surface normal against the diffuse BRDF.Source: §9 · p. 12

10. Detailed Differentiable Ambient Occlusion

The module makes screen-space ambient occlusion radius optimization differentiable while retaining the original operator's efficient CUDA implementation, with body-part-aware sampling and a two-stage training procedure.

10. Detailed Differentiable Ambient Occlusion

  • The differentiable SSAO module is designed to optimize occlusion radii while preserving the original operator's efficient CUDA implementation.Its body-part-specific radius parameters are learnable offsets assigned to surfels and rendered as a radius map.Source: §10 · p. 12, §10.1 · p. 12

10.1. Body-Part Specific Radius Parameterization

  • Nine semantic body parts receive radius parameters, which are assigned to surfels and splatted into a rendered radius map.The body-part set includes torso, hip, knee, foot, head, shoulder, elbow, wrist, and hand.Source: §10.1 · p. 12

10.2. Custom Backward Propagation

  • Custom backward propagation provides a gradient path only for body-part-specific radius parameters; perturbed branches and other SSAO input maps remain fixed.The gradient propagates through the baseline radius branch to the per-part offsets.Source: §10.2 · p. 13

10.3. Bone-Aware Direction Map

  • The bone-aware direction map blends pose-dependent joint frame axes using LBS weights, then projects the direction onto the surfel tangent plane to form a sampling basis.The resulting direction field is splatted into image space.Source: §10.3 · p. 13

10.4. Sample Importance Map

  • The sample importance map adapts SSAO density to body-part scale, allocating up to four times more sampled directions to thin appendages while leaving torso sampling unchanged.Importance is computed from joint radii and distance-weighted LBS influences, with smooth transitions that require no part segmentation.Source: §10.4 · pp. 13–14

10.5. Detailed Training Losses

  • Training first reconstructs geometry with directional color, radiance, geometry, normal-deformation, and regularization losses; the second stage estimates BRDF materials and light with shaded-color losses and added regularization.The second stage adds TV smoothing for rendered albedo, roughness, and environment light, and regularizes light toward white.Source: §10.5 · p. 14

Related papers

Loading 2609.27600v1…