PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Semantic Inpainting for Animatable Gaussian Head Avatars

Pilseo Park, Fizza Rubab, Yiying Tong

arXiv:2609.38343v1cs.CV

TL;DR

Single-image avatar reconstruction leaves unobserved facial regions ambiguous and makes identity-consistent animation challenging. SInGA completes identity-specific features in UV space using facial symmetry, then predicts and stacks Gaussians for an animatable avatar. The authors report competitive or better reconstruction quality and expression accuracy than prior approaches, alongside limitations from asymmetric hidden details and pose accuracy.

  • Problem

    Single-image head-avatar reconstruction must handle ambiguous unobserved regions and realistic, consistent facial deformations across expressions while preserving identity.

    Source: §1 · p. 1

  • Method

    SInGA maps image features to a structured UV representation, uses symmetry-guided completion for unseen regions, and regresses Gaussian attributes, with stacking to represent fine detail.

    Source: §1 · p. 2, §3.3 · p. 7, §3.4 · p. 8

  • Results

    SInGA achieves competitive or better reconstruction quality and expression accuracy than previous approaches while enabling real-time avatar animation.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The resulting avatars preserve identity-specific details in unobserved regions while supporting accurate reenactment and animation.

    Source: §1 · p. 3, Qualitative Results · p. 12

    Hidden asymmetric details remain ambiguous from one image, lighting asymmetry affects completion, and the method can have higher APD than some baselines and jaw contour drift in challenging cases.

    Source: §5 · p. 14

Abstract

from arXiv · show

We present SInGA, a novel method for learning Semantic Inpainting for animatable Gaussian head Avatars from a single image. Existing avatar approaches often rely on multi-view observations and lack effective handling of unobserved regions in single-view settings, limiting their applicability in such scenarios. To address this, we propose a semantic inpainting framework defined in UV space for completing unobserved facial regions. Our key insight lies in the structured topology of the UV representation, which provides consistent spatial correspondences and enables reliable completion of identity-specific features using the inherent symmetry cues of human faces. We extract features from observed regions and use them to complete unobserved regions. The completed representation is then used to regress Gaussian attributes, effectively performing Gaussian inpainting. In addition, instead of relying on a single Gaussian at each surface or pixel location, we stack multiple Gaussians to enhance detail. The resulting avatar generalizes across identities without requiring per-identity optimization and can be animated with driving inputs. Experimental results show that our method generates high-quality head avatars with improved completeness and identity preservation, while supporting realistic animation and consistent rendering from unobserved views.

1 Introduction

Single-image avatar reconstruction must infer unobserved facial regions and preserve identity through expression changes. SInGA uses a structured UV representation and symmetry-guided feature completion to produce animatable Gaussian head avatars, with competitive or better reconstruction quality and expression accuracy than prior approaches.

1 Introduction

  • Single-image reconstruction must infer unobserved regions from incomplete evidence and model realistic, consistent facial deformations across expressions while preserving identity.Source: §1 · p. 1
  • SInGA reconstructs a complete animatable Gaussian head avatar from one image and achieves competitive or better visual quality while enabling real-time animation.The method also reports competitive or better expression accuracy compared with previous approaches.Source: §1 · p. 3

2 Related Work

2.1 UV-Space Representation

  • UV representations encode surface attributes on regular 2D grids, providing consistent correspondence and efficient processing; SInGA uses FLAME UV correspondences to assign identity-specific semantic features to unseen regions.Source: §2.1 · pp. 3–4

2.2 Foundation Model as Prior

  • SInGA adopts DINOv2 as a semantic prior for identity-aware feature representation.Source: §2.2 · p. 4

3 Method

SInGA maps source-image features onto a fixed FLAME UV surface, completes unobserved regions using symmetry and confidence-aware aggregation, and predicts Gaussian attributes for an animatable head avatar. It adds region-aware Gaussian stacking and trains the model with image reconstruction, perceptual, and geometry regularization losses.

3.1 Feature Mapping in UV Space

  • The UV representation maps source-image features to visible surface locations, then completes missing features before Gaussian prediction.The fixed FLAME topology aligns UV coordinates with canonical surface points; DINOv2 features are sampled only where the source view provides visibility.Source: UV Feature Representation · p. 5, §3.1 · p. 5, Figure 2 · p. 5

3.2 Identity-Consistent Inpainting

  • Geometry-aware symmetry correspondence supplies opposite-side features, while confidence-weighted aggregation preserves reliable observations and learnable parameters fill remaining unobserved pixels.Confidence is lower near silhouettes and grazing angles, where direct use of mirrored features can introduce artifacts.Source: Symmetry-Guided Completion · p. 6, Confidence-Aware Aggregation · pp. 6–7

3.3 Region-Aware Gaussian Stacking

  • Region-aware stacking allocates additional Gaussians to selected complex regions while removing the same number from coarse regions, keeping the overall Gaussian budget unchanged.Retained coarse-region Gaussians are rescaled to preserve spatial coverage.Source: §3.3 · pp. 7–8

3.4 Gaussian Attribute Prediction

  • A layer-wise UV representation predicts Gaussian attributes from completed features, placing each layer by offsets along the surface normal and predicting appearance attributes from updated features.For stacked Gaussians, additional layers are placed outward from the base layer, and source-view features are resampled for appearance prediction.Source: §3.4 · p. 8

3.5 Reenactment and Refinement

  • FLAME driving parameters transfer target motion while preserving source identity, and a UNet-based neural renderer refines the rendered feature map into the final image.Only the Gaussian position map is updated during reenactment; the renderer receives a 32-channel map whose first three channels are coarse RGB.Source: §3.5 · p. 9

3.6 Training Strategy and Optimization

  • Training samples source and target frames from the same video and supervises coarse and refined outputs with L1 and perceptual losses, alongside offset regularizers.Mean, magnitude, and total-variation penalties regularize predicted offsets.Source: §3.6 · p. 9

4 Experiments

Experiments compare SInGA with feed-forward avatar methods on reconstruction, identity, expression, and runtime measures. They also assess unseen-region performance and ablate position offsets, symmetry completion, and Gaussian stacking.

4.1 Experimental Settings

4.2 Main Results

  • Across VFHQ and HDTF, SInGA reports better PSNR and SSIM, competitive LPIPS, CSIM, AKD, and AED, but worse APD; qualitative results show identity-consistent unseen-region details under pose changes.The qualitative comparisons include identity-specific hair color, hairstyle, and facial hair.Source: Table 1 · p. 11, Table 2 · p. 11, Qualitative Results · pp. 11–12

4.3 Evaluation on Unobserved Regions

  • On source-unobserved, target-visible regions, the full model improves PSNR, SSIM, and LPIPS over the no-symmetry variant and has higher PSNR and SSIM than GAGAvatar.The reported margins are larger than in full-image evaluation.Source: Table 3 · p. 12, §4.3 · p. 12

4.4 Efficiency

  • SInGA is not the fastest renderer or quickest reconstruction method, but it maintains short reconstruction time while supporting real-time rendering.Reported runtimes exclude source preprocessing and FLAME parameter estimation.Source: §4.4 · p. 12, Table 4 · p. 13, Table 5 · p. 13

4.5 Ablation Studies

  • In eye-region self-reenactment, Gaussian stacking reduces iris–sclera color bleeding and blurred eye corners, yielding a cleaner eyelid boundary and finer structural detail.The stacking metrics are computed over the eye region.Source: Table 7 · p. 13, Figure 7 · p. 14, Gaussian Stacking · p. 14
  • Predicted position offsets improve detail reconstruction by enabling more accurate UV feature sampling than keeping Gaussians at their initial FLAME-surface positions.The ablation evaluates removing the offset used to compute final Gaussian positions.Source: Position Offset · p. 13, Table 6 · p. 13
  • Symmetry completion transfers information from visible corresponding regions, producing more reliable identity-consistent reconstructions than learnable features for unseen areas.The comparison is shown qualitatively and supported by the reported quantitative ablation.Source: Symmetry Completion · p. 13, Table 6 · p. 13, Figure 6 · p. 14

5 Conclusion and Limitations

SInGA predicts animatable Gaussian head avatars from one image, using UV-aligned structure for semantic inpainting and Gaussian stacking. Its completion remains limited by hidden asymmetry and lighting, with additional challenges in APD and jaw contours.

5 Conclusion and Limitations

  • SInGA predicts an animatable Gaussian head avatar from one image using a UV-aligned surface representation.Fixed topology supports semantically guided inpainting of unobserved regions and region-adaptive enhancement.Source: §5 · p. 14
  • Symmetry-guided completion transfers information from visible corresponding regions to recover more identity-consistent features.Gaussian stacking increases representation capacity in selected regions to capture fine details.Source: §5 · p. 14
  • Hidden asymmetric details remain ambiguous from one image, and lighting asymmetry affects symmetry-guided completion.The method also has higher APD than some baselines and can show jaw contour drift in challenging cases.Source: §5 · p. 14

6 Preliminaries

The preliminaries describe FLAME as an animatable facial-geometry prior and 3D Gaussian Splatting as an explicit representation rendered through differentiable rasterization and compositing.

6 Preliminaries

  • FLAME deforms a template mesh using identity shape, expression, and pose blendshapes, then obtains posed vertices through linear blend skinning.The posed mesh uses articulated joint locations and skinning weights; FLAME provides the method's structured head prior.Source: §6 · p. 19
  • A 3D Gaussian is parameterized by its center, scale, rotation, opacity, and appearance coefficients.Its covariance is constructed from scale and rotation.Source: §6 · p. 19
  • Differentiable tile-based rasterization and alpha compositing enable efficient, high-fidelity rendering of explicit 3D Gaussian primitives.Source: §6 · p. 19

7 Network Architecture

The geometry and appearance networks consume different feature maps, with viewing-direction encoding appended to both. Their prediction heads produce Gaussian offsets and attributes, including multiple outputs for stacked regions.

7 Network Architecture

  • The geometry network takes the aggregated feature map as input, while the appearance network operates on the resampled feature map.Both feature maps have size 256 × 512 × 512.Source: §7 · p. 20
  • Both networks concatenate a 27-dimensional harmonic encoding of viewing direction with their input feature maps.Source: §7 · p. 20
  • For K-layer stacked regions, prediction heads output K × 1 geometry offsets and K × 40 Gaussian attributes.Each attribute vector contains 32 color channels, opacity, three scale parameters, and four rotation parameters; the architecture tables cover the geometry and appearance networks.Source: §7 · p. 20, Table 8 · p. 20, Table 9 · p. 20

8 Additional Qualitative Results

Additional qualitative results cover cross- and self-reenactment on VFHQ and HDTF, plus in-the-wild images. The authors report identity preservation and 3D consistency across viewpoints in the shown results.

8 Additional Qualitative Results

  • The authors report that their method transfers driving expressions while preserving fine identity details in self-reenactment cases with small source-to-driving pose differences.Source: §8 · p. 20
  • Cross-reenactment comparisons on VFHQ and HDTF are shown in Fig. 8.Source: Figure 8 · p. 21
  • Self-reenactment comparisons on VFHQ and HDTF are shown in Fig. 9.Source: Figure 9 · p. 22
  • In-the-wild examples show source images in the first column, driving images in the top row, and generated results in the remaining images.The authors describe these results as generalizing while preserving identity and 3D consistency across viewpoints.Source: §8 · p. 20, Figure 10 · p. 23

9 Dataset Licenses and Ethical Considerations

Experiments use VFHQ and HDTF, both released for non-commercial research, with additional in-the-wild images for qualitative generalization evaluation. No additional personally identifiable information is used beyond the facial images required for the task.

9 Dataset Licenses and Ethical Considerations

  • Experiments use the publicly available VFHQ and HDTF datasets, which are released for non-commercial research use.Source: §9 · p. 24
  • In-the-wild images are included for qualitative evaluation of generalization under unconstrained conditions.Source: §9 · p. 24
  • The experiments use no additional personally identifiable information beyond the facial images required for the task.Source: §9 · p. 24

10 Potential Societal Impacts

The authors note that photorealistic head avatars could be misused to create deceptive manipulated media involving real individuals and do not support or encourage such applications. They suggest avatar-generated training data as one possible countermeasure for deepfake detection models.

10 Potential Societal Impacts

  • Photorealistic human head avatars could be misused to create deceptive manipulated media involving real individuals.Source: §10 · p. 24
  • The authors state that they do not support or encourage such applications.Source: §10 · p. 24
  • One possible countermeasure is training deepfake detection models with generated avatars to better separate avatar renderings from real images.Source: §10 · p. 24

Related papers

Loading 2609.38343v1…