PaperlayerView exact revision on arXiv

Source-linked AI summary

Avat3r: Large Animatable Gaussian Reconstruction Model for High-fidelity 3D Head Avatars

Tobias Kirschstein, Javier Romero, Artem Sevastopolsky, Matthias Nießner, Shunsuke Saito

arXiv:2502.20220v2cs.CV

TL;DR

Existing high-fidelity avatar creation often depends on studio capture and expensive optimization, limiting sparse casual use. Avat3r regresses animatable 3D head avatars from a few images using foundation-model cues, expression-conditioned attention, and confidence-filtered Gaussian prediction. The authors report strong few-input and single-input applicability, including generalization to antique busts and other out-of-distribution inputs.

  • Problem

    High-fidelity 3D head avatars traditionally require studio-level multi-view capture and expensive test-time optimization, limiting use from casual few-image captures.

  • Method

    Avat3r combines DUSt3R position maps, Sapiens features, per-pixel Gaussian prediction, and expression-conditioned cross-attention to reconstruct animatable head avatars.

  • Results

    Avat3r creates animatable avatars from a few or single images and generalizes facial movement to out-of-distribution inputs such as antique busts and AI-generated images.

  • Takeaways & Limitations

    The results support using Large Reconstruction Model-like architectures to infer animatable 3D head representations from sparse image inputs.

  • Takeaways & Limitations

    Single-image inference relies on a 3D GAN that can lose details and introduce view inconsistencies through screen-space super-resolution.

Abstract

from arXiv · show

Traditionally, creating photo-realistic 3D head avatars requires a studio-level multi-view capture setup and expensive optimization during test-time, limiting the use of digital human doubles to the VFX industry or offline renderings. To address this shortcoming, we present Avat3r, which regresses a high-quality and animatable 3D head avatar from just a few input images, vastly reducing compute requirements during inference. More specifically, we make Large Reconstruction Models animatable and learn a powerful prior over 3D human heads from a large multi-view video dataset. For better 3D head reconstructions, we employ position maps from DUSt3R and generalized feature maps from the human foundation model Sapiens. To animate the 3D head, our key discovery is that simple cross-attention to an expression code is already sufficient. Finally, we increase robustness by feeding input images with different expressions to our model during training, enabling the reconstruction of 3D head avatars from inconsistent inputs, e.g., an imperfect phone capture with accidental movement, or frames from a monocular video. We compare Avat3r with current state-of-the-art methods for few-input and single-input scenarios, and find that our method has a competitive advantage in both tasks. Finally, we demonstrate the wide applicability of our proposed model, creating 3D head avatars from images of different sources, smartphone captures, single images, and even out-of-domain inputs like antique busts. Project website: https://tobias-kirschstein.github.io/avat3r/

1. Introduction

Avat3r addresses the challenge of reconstructing animatable, high-fidelity 3D head avatars from only a few potentially inconsistent images. It combines a learned expression prior with pixel-based Gaussian prediction and foundation-model features for sparse reconstruction.

  • Few-image avatar creation is motivated by applications requiring convincing digital doubles without studio-quality, hour-long capture sessions.
  • Avat3r reconstructs an animatable 3D head avatar from just a few images of a person.
  • Sparse reconstruction is underconstrained because available views may omit regions such as the mouth interior or side of the head.
  • Animating an unseen person requires plausible facial motion despite not observing that individual performing every expression.
  • Casual captures can contain inconsistent images because the person may move between frames, creating an additional reconstruction challenge.
  • The method predicts Gaussians per foreground pixel rather than using a fixed template, while DUSt3R positions and Sapiens features simplify sparse reconstruction.

2. Related Work

Related work spans sparse-view 3D reconstruction, monocular avatar animation, and learned 3D face priors, but prior approaches leave animatable sparse-input reconstruction insufficiently addressed. Avat3r targets this gap while avoiding broad identity generalization.

  • 2.1. Large 3D Reconstruction Models: Large Reconstruction Models infer 3D representations from sparse images, but existing works had not studied animatable 3D representations from sparse input images.
  • Monocular avatar methods use meshes, NeRFs, SDFs, points, or Gaussians, but optimization can overfit the input viewpoint and weaken viewpoint extrapolation.
  • 3D-aware portrait-animation methods inject control through representations such as 3D morphable models, TriPlanes, and 3D Gaussians.
  • Photorealistic 3D face models learn geometry and appearance priors, but training-subject counts rarely exceed thousands, limiting coverage of human facial appearance.
  • Codec avatars can achieve excellent quality but may require minutes of capture and expensive test-time optimization, including a reported three-hour finetuning process on eight A100 GPUs for URAvatar.

3. Method

Avat3r combines foundation-model geometry and semantic features with a Vision Transformer that predicts per-pixel Gaussian attributes and conditions them on expression codes. Confidence filtering produces a renderable Gaussian set from posed input images.

  • 3.1. Pre-trained Foundation Models: Avat3r predicts a high-quality 3D Gaussian representation from a few posed images while simultaneously matching a desired facial expression.
  • 3.1. Pre-trained Foundation Models: DUSt3R supplies position and confidence maps, while Sapiens supplies semantic feature maps that support Gaussian placement and cross-view matching.
  • 3.2. Animatable Large 3D Reconstruction Model: Input images and foundation-model maps are patchified into tokens, then dense self-attention performs within-image and cross-view matching to infer 3D structure.
  • 3.2. Animatable Large 3D Reconstruction Model: Cross-attention lets image tokens attend to a sequence derived from the expression code, and this simple conditioning models complex facial animations in 3D.
  • 3.2. Animatable Large 3D Reconstruction Model: The decoder upsamples intermediate features into per-pixel positions, scales, rotations, colors, and opacities, with skip connections for positions and colors.
  • 3.2. Animatable Large 3D Reconstruction Model: Only pixels whose DUSt3R confidence exceeds τ = 0.5 spawn Gaussians, reducing initialization artifacts and adapting the primitive count to the subject.
  • 3.2. Animatable Large 3D Reconstruction Model: The retained Gaussians can be rendered from arbitrary viewpoints using a tile-based differentiable rasterizer.
  • Training uses photometric novel-view losses with perceptual losses for high-frequency details, weighted by λ_l1 = 0.8, λ_ssim = 0.2, and λ_lpips = 0.01.

4. Experimental Results

Avat3r is evaluated for few-shot 3D head avatar creation, ablations, expression generalization, flexible view counts, and application scenarios. The results show strong quality, robustness to inconsistent inputs, expression interpolation, and applicability beyond the training setting, while single-image inference remains limited by 3D lifting errors.

  • Few-shot avatar creation: Avat3r produces realistic avatars on NeRSemble despite that dataset being unseen during training and differing in lighting, cameras, and viewpoints.The comparison uses four input images and evaluates out-of-domain source and driver persons.
  • Few-shot avatar creation: Avat3r creates realistic animatable head avatars from four images and outperforms the baselines across all reported metrics.Evaluation uses PSNR, SSIM, LPIPS, AKD, and CSIM on held-out subjects.
  • Ablations: DUSt3R position maps improve geometric alignment, while Sapiens features improve sharpness, especially around hair.Removing DUSt3R impairs alignment; Sapiens contributes semantic information for sharper predictions.
  • Ablations: Training with inconsistent input images improves robustness, whereas training only on temporally consistent views can transfer input inconsistencies into artifacts.The final model balances alignment, sharpness, and robustness rather than optimizing sharpness alone.
  • Additional results: The model generalizes between extreme expressions for unseen source and driver persons and accepts more or fewer than four input views.Expression interpolation is demonstrated separately from direct inference with arbitrary view counts.
  • Applications and limitations: Single-image and out-of-domain applications are demonstrated, but single-image inference accumulates 3D-lifting errors that reduce final-avatar quality.The pipeline uses a 3D GAN to lift one image and can animate generated, text-to-image, or photographed heads.

5. Conclusion

Avat3r directly regresses animatable 3D head avatars from four images, combining large reconstruction models with DUSt3R, Sapiens, and cross-attention for dynamics. It also supports single-image inputs through a pre-trained 3D GAN and generalizes to out-of-distribution examples.

  • Avat3r directly regresses an animatable 3D head avatar from just four input images.
  • The pipeline combines large reconstruction models with DUSt3R and Sapiens, while modeling facial dynamics with simple cross-attention layers.
  • A pre-trained 3D GAN enables Avat3r to operate in a single-image scenario.
  • Avat3r learns a facial-movement prior that generalizes to out-of-distribution inputs such as antique busts and AI-generated images.
  • The architecture may support future training on large monocular-video datasets and use as a 3D lifting and denoising module in diffusion frameworks.

A. Additional Results

Additional phone scans show that Avat3r handles varied capture conditions and challenging appearance factors, including glasses, rotated inputs, and headscarves.

  • Avat3r reconstructs phone scans captured with different devices, including cases with glasses, rotated inputs, and a headscarf.
  • Despite not being trained on accessories, Avat3r handles glasses reasonably well and reconstructs the headscarf with high fidelity.

A.2. Single-shot 3D Head Avatar Creation

Avat3r can create animatable 3D head avatars from single images by lifting them into four rendered views before reconstruction. Qualitative comparisons report competitive performance against several single-input portrait-animation baselines.

  • A pre-trained 3D GAN lifts one input image into 3D and renders four views that Avat3r uses for reconstruction.
  • The single-image comparison includes GAGAvatar, Portrait4Dv2, and HeadNeRF as portrait-animation baselines.
  • Avat3r performs competitively against single-input baselines despite never being trained specifically for a single-shot scenario.

A.3. Comparison with Monocular Methods

Comparisons with monocular and single-image methods examine generalization, data efficiency, and input-view requirements across several datasets and ablations. The reported analyses show that reconstruction quality depends on training diversity and multiple input views, while monocular methods can overfit to the training view.

  • Comparison with Monocular Methods: FlashAvatar performs well from its training view but fails on novel views because of overfitting, whereas Avat3r produces a plausible 3D head avatar.
  • Single-image Comparison: The single-image comparison on Ava256 reports better rendering quality for Avat3r with 3D lifting, especially under extreme expressions.
  • Effect of Number of Train Subjects: Training on a larger and more diverse set of people improves Avat3r’s generalization and identity matching, including complex hairstyles.
  • Effect of Number of Input Views: Training with a single input image noticeably impairs quality, while using more than four input images yields no significant improvement.
  • Identity Preservation: Adding 984 identities with neutral expressions improves identity preservation.

B. Analysis of Data Efficiency

Avat3r improves as both training subjects and input views increase, but adding views substantially raises runtime and rendering costs.

  • More training subjects improve Avat3r’s performance on the Ava256 dataset.
  • More input views also improve performance when reconstructing avatars.
  • Increasing input views drastically increases runtime because transformer attention becomes denser and more Gaussians must be rendered.

B.2. Effect of Adding More Neutral Subjects

Adding identities with only one expression improves identity retention, addressing overfitting risk without requiring many expressions per added subject.

  • Avat3r risks identity overfitting because it is trained on only 244 Ava256 subjects.
  • Adding 984 identities with one expression each increases training data by only 0.08%.
  • The added identities noticeably improve identity retention for challenging cases, confirmed by improved CSIM metrics.

E. Training Details

Training uses processed head-centered data, precomputed foundation-model features, controlled viewpoints and expressions, and canonical head coordinates.

  • Dataset processing: Ava256 processing uses foreground masks, black backgrounds, and head-centered 512 × 512 input crops with 667 × 667 supervision crops.
  • DUSt3R and Sapiens: DUSt3R position maps and Sapiens feature maps are precomputed for training inputs, using three reasonable partner views per input.
  • Head-centric coordinates: Factoring tracked head poses into camera poses lets Avat3r predict every head in canonical pose while excluding torso motion.
  • Expression codes: Expression codes provide driving signals, and the model learns a general facial-expression notion adaptable to different signals.
  • Sampling: Training samples spread-out camera views and different expression timesteps to improve input diversity and robustness to inconsistencies.
Loading 2502.20220v2…