PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialAvatar-0: High-Quality 4D Head Avatar with Multi-Stage Reconstruction

Yiran Wang, Zeyu Zhang, Yuanming Li, Ziming Wang, Yang Zhao

arXiv:2606.15659v1cs.CV

TL;DR

Existing 4D head-avatar systems separate feed-forward prediction from costly per-subject refinement, limiting their ability to share a common representation. SpatialAvatar-0 unifies both regimes with a shared FLAME-mesh-bound Gaussian pipeline, achieving stronger cross-domain and per-subject results with substantially shorter refinement.

  • Problem

    Existing feed-forward predictors use fixed source counts and single-dataset training, while per-subject refiners are costly and disrupt Gaussian layouts, hindering a shared representation.

    Source: §1 · pp. 1–2

  • Method

    SpatialAvatar-0 uses a shared FLAME-mesh-bound Gaussian representation with variable-source feed-forward generation, two-phase training, and 10K-iteration layout-preserving refinement.

    Source: §1 · p. 2, §3.1 · pp. 3–4

  • Results

    +1.5 dB PSNR over GAGAvatar on zero-shot VFHQ/HDTF and +1.3 dB PSNR over 300K-iteration GeoAvatar while leading every reported SplattingAvatar metric.

    Source: §4.3 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    The results demonstrate that generalizable feed-forward and per-subject 3DGS avatar regimes can be developed within one coherent pipeline.

    Source: §5 · p. 9

    Per-subject refinement remains approximately two minutes, so true real-time avatar creation is limited to the feed-forward stage.

    Source: C Limitations and Future Work · p. 24

Abstract

from arXiv · show

High-quality 4D head avatars from one or a few source portraits are central to telepresence, AR/VR, and digital-human interaction. 3D Gaussian Splatting (3DGS) has emerged as the dominant representation, with two complementary regimes (generalizable feed-forward predictors and per-subject refiners) maturing in parallel. However, existing feed-forward predictors are trained on a single dataset family with a hard-coded source count, inheriting the corresponding domain bias. Per-subject refiners require 300K--600K iterations and rely on adaptive densification that destroys upstream Gaussian layouts, preventing the two regimes from sharing a representation end-to-end. To bridge both regimes we propose SpatialAvatar-0 on a shared FLAME-mesh-bound Gaussian representation: a feed-forward generator with a parameter-free K-source mean-pool and a monocular-temporal to multi-view-spatial two-phase schedule that anchors against identity-prior collapse onto the smaller multi-view set. We further introduce a 10K-iter layout-preserving per-subject refinement loop that freezes the FLAME-binding and Gaussian count and replaces densification with a three-component anti-spike regularization. On VFHQ/HDTF cross-domain zero-shot we surpass the in-domain leader GAGAvatar by +1.5 dB PSNR despite never training on either test domain, and on the SplattingAvatar monocular benchmark we lead every reported metric, surpassing the 300K-iter GeoAvatar by +1.3 dB PSNR at up to 60x shorter per-subject schedule than common SOTA baselines. Website: https://spatialwalk.github.io/SpatialAvatar-0.

1 Introduction

SpatialAvatar-0 unifies feed-forward prediction and per-subject refinement through a shared FLAME-mesh-bound Gaussian representation. It addresses dataset and source-count constraints in feed-forward models while enabling faster, layout-preserving refinement without densification.

Motivation

  • 4D head avatars support telepresence, AR/VR communication, and digital-human interaction, motivating both generalizable feed-forward predictors and per-subject 3DGS refiners.Feed-forward predictors generate animatable face-bound Gaussian models from one or a few portraits, while refiners optimize individual subjects.Source: §1 · p. 1

Problem

  • Existing feed-forward avatars inherit single-dataset domain bias and hard-code source-image count, limiting multiview geometry and variable-source deployment.Monocular training lacks ground-truth multiview geometry, while fixed source counts prevent exploiting arbitrary numbers of available views.Source: §1 · p. 1
  • 300K–600K iterations per identity and adaptive densification make per-subject refinement slow and destroy feed-forward spatial inductive bias, while removing densification causes anisotropic Gaussian overfitting.Without densification, Gaussians can elongate along one axis and catastrophically break novel-view rendering.Source: §1 · p. 2

Method

  • SpatialAvatar-0 uses a shared FLAME-mesh-bound Gaussian representation, a parameter-free K-source mean-pool, and monocular-temporal →multi-view-spatial training to combine identity coverage with geometric grounding.The schedule uses L2-SP anchoring and a 25% NeRSemble cross-time mix to prevent identity-prior collapse onto the smaller multi-view set.Source: §1 · p. 2
  • 10K-iter layout-preserving refinement replaces densification with three-component anti-spike regularization and leads every reported SplattingAvatar metric at up to 60× shorter schedules.The refinement loop freezes the inherited Gaussian layout while avoiding the long schedules of common per-subject baselines.Source: §1 · p. 2

2 Related Work

Related work progresses from 2D talking-head synthesis toward 3D-aware representations, culminating in 3D Gaussian Splatting for efficient, geometrically faithful head avatars. SpatialAvatar-0 uses a hybrid representation with multi-scale fusion and enhanced rigging to support view-consistent rendering without per-identity optimization.

Overview

  • Head-avatar research targets photorealistic, animatable reconstruction from sparse inputs while controlling facial expressions, gaze, and head poses.Applications include telepresence, gaming, and the metaverse.Source: §2 · p. 2

2D Methods

  • Early talking-head methods used 2D generative models, feature modulation, latent manipulation, and dense warps to synthesize or deform portrait sequences.3D Morphable Models were added as low-dimensional geometric priors to improve deformation realism and structural integrity.Source: §2.1 · pp. 2–3

3D-Aware Representations

  • 3D-aware approaches emerged to address view consistency, progressing from mesh-based statistical face priors to flexible, topology-agnostic NeRF reconstruction.These methods reconstruct personalized avatars from monocular or sparse inputs.Source: §2.2 · p. 3

3D Gaussian Splatting

  • 3D Gaussian Splatting balances rendering efficiency and geometric fidelity through explicit anisotropic primitives and has been extended to head modeling with mesh or deformation-field bindings.Its tile-based rasterization enables real-time rendering.Source: §2.3 · p. 3

SpatialAvatar-0

  • In contrast, SpatialAvatar-0 combines multi-scale feature fusion with enhanced rigging for high-fidelity, view-consistent rendering without time-consuming per-identity optimization.The hybrid representation bypasses redundant volumetric-rendering computations while maintaining generalization capabilities.Source: §2.3 · p. 3

3 Method

SpatialAvatar-0 reconstructs heads in two coupled stages over a shared FLAME-mesh-bound Gaussian representation: variable-source feed-forward generation followed by optional 10K-iteration per-subject refinement. Its method combines two-phase monocular-to-multi-view training with fixed UV-pixel Gaussian layouts, frozen bindings, and anti-spike regularization.

Two-stage reconstruction

  • The pipeline uses a shared FLAME-mesh-bound 3D Gaussian representation with a feed-forward generator and optional per-subject photometric refinement.The generator accepts K ∈ {1, 2, 3, 4} source portraits, while refinement starts from its output for a chosen reference frame.Source: §3.1 · pp. 3–4

Two-phase training

  • Training proceeds from monocular-temporal CelebV-HQ pretraining to multi-view-spatial NeRSemble post-training without architectural changes.Both phases expose the model to variable K ∈ {1, 2, 3, 4} source contexts; Phase 2 samples synchronized cameras across views and time.Source: §3.1 · p. 4, §3.2 · p. 5

Gaussian representation

  • Each identity contains approximately 58K Gaussians, one per valid pixel on a 256×256 FLAME UV grid, rigidly bound to parent triangles.The UV-pixel organization is fixed infrastructure, and the binding is preserved during refinement.Source: §3.1 · p. 4, §3.2 · p. 5

Architecture

  • The architecture combines frozen DINOv3 features, UV-space StyleUNet generation, target-camera harmonic encoding, and FLAME-conditioned residual corrections.The residual head predicts per-attribute corrections for position, rotation, scale, and color, while shape β enters through FLAME geometry.Source: §3.2 · p. 4

Per-subject refinement

  • The 10K-iteration refinement freezes the FLAME mesh and triangle bindings, preserving the one-to-one UV-pixel-to-Gaussian map while optimizing against randomly sampled target frames.The mesh is deformed to each target pose, transformed Gaussians are rendered, and photometric loss is optimized.Source: §3.3 · p. 5
  • Anti-spike regularization combines a 500-iteration scale-freeze warmup, a hard log-scale clamp, and a soft screen-space anti-anisotropy penalty.These mechanisms address Gaussian elongation along a single axis when densification is removed.Source: §3.3 · p. 6

4 Experiments

Experiments evaluate SpatialAvatar-0 in feed-forward zero-shot and per-subject settings across out-of-distribution and monocular benchmarks, with ablations on held-out CelebV-HQ. The method achieves leading image-quality results while substantially reducing per-subject training cost.

Evaluation setup

  • The evaluation covers VFHQ and HDTF for feed-forward zero-shot reenactment, SplattingAvatar for per-subject evaluation, and held-out CelebV-HQ for ablations.Metrics include PSNR, SSIM, LPIPS, and feed-forward identity, expression, pose, and keypoint measures.Source: §4.1 · p. 6

Feed-forward comparison

  • Ours-FF leads every reported column on VFHQ and HDTF, surpassing the in-domain leader GAGAvatar by +1.5 dB PSNR on each split.The test domains are out-of-distribution for SpatialAvatar-0, whereas several baselines include one or both datasets during training.Source: §4.3 · pp. 6–7
  • Ours-FF preserves high-frequency identity cues, including forehead wrinkles, malar elevation, beard texture, nasolabial folds, and hair-edge silhouette across identities and poses.These qualitative findings complement the quantitative gains across the reported feed-forward metrics.Source: §4.3 · p. 7

Per-subject comparison

  • Ours+S3 leads every reported SplattingAvatar metric and surpasses GeoAvatar by +1.3 dB PSNR at 10K versus 300K iterations.The comparison covers MSE, PSNR, SSIM, and LPIPS, with GeoAvatar previously leading PSNR, SSIM, and MSE.Source: §4.3 · p. 7

Per-subject efficiency

  • ∼2 min per subject is two orders of magnitude shorter than FlashAvatar, GeoAvatar, and GaussianAvatars on a single RTX 3090.The 10K-iteration schedule is 15×, 30×, and 60× shorter than those baselines, respectively.Source: §4.3 · p. 8

Pipeline and architecture ablation

  • The headline ablation uses self-reenactment on a held-out CelebV-HQ slice, cumulatively builds the pipeline, and removes architectural components from the full configuration.Adding Phase-2 NeRSemble v2 post-training without controls catastrophically forgets CelebV-HQ; the L2-SP anchor and 25% cross-time mix are then reintroduced.Source: §4.4 · p. 9

5 Conclusion · A Additional method details

SpatialAvatar-0 bridges feed-forward and per-subject 3DGS head-avatar regimes with a shared FLAME-mesh-bound Gaussian representation. Its variable-source generator, two-phase schedule, and 10K-iteration layout-preserving refinement target cross-domain identity priors and efficient personalization.

5 Conclusion

  • SpatialAvatar-0 unifies generalizable feed-forward and per-subject 3DGS head-avatar regimes through a shared FLAME-mesh-bound Gaussian representation.Source: §5 · p. 9
  • The feed-forward generator uses a K-source-variable design and a monocular →multi-view two-phase schedule.Source: §5 · p. 9
  • +1.5 dB PSNR is achieved on zero-shot VFHQ/HDTF benchmarks over the in-domain leader.Source: §5 · p. 9
  • The two-phase generator produces a strong cross-domain identity prior for zero-shot evaluation.Source: §5 · p. 9
  • Per-subject personalization uses a 10K-iter layout-preserving refinement procedure.Source: §5 · p. 9
  • The refinement procedure is enabled by mechanisms described in the continuation of the conclusion passage.Source: §5 · p. 9

A Additional method details

A.1 Feed-forward representation and architecture

SpatialAvatar-0 uses a topology-respecting UV-aligned Gaussian representation whose mesh-bound structure supports convolutional generation with content-independent Gaussian count. Parameter-free barycentric warping, per-triangle local frames, harmonic view encoding, and expression-conditioned residual FiLM heads provide the inputs and attribute refinements for feed-forward prediction.

UV-aligned representation

  • The UV-aligned representation maps neighboring UV pixels to neighboring mesh triangles, transfers UV smoothness to the mesh surface, and fixes Gaussian count independently of image content.Its topology-respecting unwrap gives 2D convolutions an inductive bias for face-bound primitives.Source: §A.1 · p. 15

Barycentric feature warp

  • A parameter-free barycentric warp identifies each UV pixel’s FLAME triangle, interpolates its source-frame 3D position, projects it through the source camera, and samples encoder features and RGB.The warp uses precomputed triangle assignments, barycentric weights, and bilinear sampling.Source: §A.1 · p. 15

Harmonic view-direction encoding

  • The target camera direction is unit-normalized and represented with a 27-dimensional harmonic embedding for view-dependent conditioning.The direction is taken from the third column of the target view’s world-to-camera matrix.Source: §A.1 · p. 15

Residual head conditioning

  • A 2-layer MLP generates per-attribute FiLM coefficients for position, rotation, scale, and color, whose 1 × 1 convolutional outputs refine the decoded UV maps while excluding opacity.The residual head conditions on target expression, pose, and eye codes; shape is excluded because it is fixed per subject and directly feeds mesh geometry.Source: §A.1 · p. 15

Per-triangle frame construction

  • Per-triangle local frames use the centroid for translation, Gram-Schmidt-like edge-and-normal orthonormalization for rotation, and averaged edge-length and altitude geometry for isotropic scale.These frames define the local basis for Gaussian attributes and quaternion corrections.Source: §A.1 · p. 16

A.2 Training procedure

Training uses difficulty-weighted source sampling and a two-phase schedule that combines near-uniform early exploration with predominantly cross-camera multi-view supervision in Phase 2. The procedure specifies explicit loss handling and distinct optimization settings across phases.

Source sampling

  • Source frames are sampled using a softmax over camera-direction and expression-code dissimilarity to avoid trivial near-identity source-target pairs.The softmax temperature is annealed from high, approximately uniform sampling toward low-temperature focused sampling during early training.Source: §A.2 · p. 16

Phase 2 sampling

  • Phase 2 samples K+1 cameras from synchronized 16-camera NeRSemble captures, using 75% cross-camera batches and 25% cross-time batches by default.Cross-camera draws provide multi-view supervision, while cross-time draws preserve temporal variation; when K=1, the latter reduce to monocular same-camera-different-time sampling.Source: §A.2 · p. 16

Loss formulation

  • All UV-domain loss terms are averaged over the valid-UV mask, while TV penalties operate on the UV grid and can smooth across FLAME UV seams.The valid mask contains approximately 58,173 pixels, and the cross-seam contribution is described as small in this setting.Source: §A.2 · p. 17

Optimization

  • Phase 1 uses Adam at 2.5×10−4 with FP32, batch size 48, and gradient clipping at norm 5.0; Phase 2 uses AdamW with cosine warmup, bf16, and batch size 12.Phase 2 additionally sets λsp=10−3, with a 25% NeRSemble cross-time mix share; Lδ is inactive in Phase 1.Source: §A.2 · p. 17

A.3 Jacobian penalty derivation … B.3 Backbone capacity and source-image count

The paper derives an efficient Hutchinson-based Jacobian regularizer for view-direction invariance, specifies refinement safeguards and initialization details, and reports qualitative evidence across monocular, cross-identity, backbone, and source-count settings. The ablations show sharper reconstructions, monotonic backbone gains, and diminishing returns beyond two source images.

A.3 Jacobian penalty derivation

  • The regularizer drives position and scale Jacobians with respect to view encoding γ(d) toward zero at every UV pixel, while leaving color and opacity view-dependent.Rotation is excluded because raw quaternion-component penalties do not represent angular-Jacobian energy under the unit-sphere constraint.Source: §A.3 · pp. 17–18
  • A Hutchinson per-pixel estimator avoids the one-to-two-orders-of-magnitude autograd overhead of explicit per-pixel Jacobian evaluation.The finite-difference alternative is rejected because moderate-to-large angular separations confound first-order sensitivity with higher-order curvature.Source: §A.3 · pp. 17–18
  • The log(1 + ·) wrapper caps early-training regularizer magnitude, when the unwrapped argument is approximately 10^2, while approaching identity as the argument shrinks.The wrapped objective incurs a second-order Jensen bias that decreases as the expected Jacobian energy approaches zero.Source: §A.3 · p. 18

A.4 Per-subject optimization details

  • Per-subject refinement combines a hard parameter-space log-scale clamp with a soft screen-space projected-aspect penalty, enforcing world-space anisotropy bounds while encouraging isotropic footprints.The two controls act on different quantities and are therefore not redundant.Source: §A.4 · p. 19
  • Reference-frame initialization selects the minimum jaw-pose magnitude, but falls back to a random bottom-decile frame when the minimum is below τjaw = 10^-3.The heuristic is jaw-only and does not guarantee a fully canonical pose.Source: §A.4 · p. 19

B.1 Per-subject monocular qualitative comparison

  • Ours+S3 follows held-out target poses closely without visible blur or dropped regions, preserving wrinkles, facial contours, hair edges, and eye highlights at rendered scale.The visualization sample contains four subjects and compares Ours+S3 with five per-subject baselines.Source: §B.1 · p. 20, Figure 3 · p. 20

B.2 Cross-identity feed-forward qualitative comparison

  • Ours-FF retargets source identity under substantially different driver yaw and expression while preserving source-consistent eye contours, nose-bridge orientation, and jaw outline.The comparison covers VFHQ and HDTF test splits against GAGAvatar, CVTHead, GPAvatar, and Portrait4D-v2.Source: §B.2 · p. 20, Figure 4 · p. 21

B.3 Backbone capacity and source-image count

  • DINOv3 backbone quality improves monotonically from S to L, with DINOv3-B near the quality–parameter knee, while source-count gains are largest from K=1 to K=2 and flatten thereafter.The encoder is frozen, and the source-count ablation evaluates K ∈ {1, 2, 3, 4} at inference time.Source: §B.3 · pp. 21–22, Table 6 · p. 22

B.4 Training hyperparameter ablation

This ablation varies four feed-forward training hyperparameters while holding the others at default settings. LinearLR and a Phase 1 base learning rate near 2.5×10−4 are most influential, whereas Phase 2 layer-wise decay is comparatively insensitive on PSNR.

Ablation setup

  • The ablation varies the Phase 1 learning-rate schedule, Phase 1 base learning rate, Phase 2 encoder layer-wise LR decay γLR, and Jacobian penalty weight λjac independently.Row (a) reproduces Tab. 5 row (C), while Default rows (e), (h), and (k) are independent reruns.Source: §B.4 · p. 22

Phase 1 schedule

  • ∼0.1–0.5 dB PSNR: LinearLR outperforms cosine and constant Phase 1 schedules, with constant LR performing worst.These comparisons correspond to rows (a)–(c).Source: §B.4 · p. 22

Phase 1 base learning rate

  • 2.5×10−4: the Phase 1 base learning rate forms a single PSNR peak, while halving or doubling it loses ∼0.2–0.5 dB.The comparison uses rows (d)–(f).Source: §B.4 · p. 22

Phase 2 layer-wise decay

  • Under 0.2 dB: deviations in either direction for Phase 2 layer-wise decay γLR regress PSNR, making it the least sensitive setting.The L2-SP anchor and cross-time mix provide the Phase 2 cross-domain protection.Source: §B.4 · p. 22

B.5 L2-SP anchor and cross-time mix ablation

The ablation evaluates the L2-SP anchor and NeRSemble cross-time mixing in Phase 2, finding complementary cross-domain protection and sensitivity to anchor strength. The adopted setting is λsp=10−3 with 25% cross-time mix.

Ablation design

  • The 2×2 factorial isolates λsp ∈{0, 10−3} and cross-time mix share∈{0%, 25%} to measure each Phase 2 control individually and jointly.Rows (a)–(d) form the factorial design, while rows (e)–(f) vary λsp at the chosen mix share.Source: §B.5 · p. 23

Complementarity

  • The anchor alone and cross-time mix alone each provide partial cross-domain protection, whereas their joint configuration recovers the full effect.This indicates the two mechanisms are complementary rather than redundant.Source: §B.5 · p. 23

Anchor sensitivity

  • λsp=10−3 with 25% cross-time mix is adopted because under-anchoring approaches cross-time-mix-only performance, while over-anchoring limits multi-view signal absorption.The sensitivity comparison uses λsp∈{10−2, 10−4} at the chosen mix share.Source: §B.5 · p. 23

B.6 Anti-spike component ablation

The anti-spike regularization combines a scale-freeze warmup, post-500 log-scale clamp, and screen-space anti-anisotropy penalty. Ablations show each component blocks a distinct failure mode that the remaining components cannot fully correct.

B.6 Anti-spike component ablation

  • Tab. 9 compares the full anti-spike configuration with variants removing the scale-freeze warmup, log-scale clamp, or Laspect while keeping the other two components fixed.The warmup spans iterations 0–500; the clamp and Laspect activate from iteration 500.Source: §B.6 · p. 23
  • Removing the scale-freeze warmup restores early anisotropy spikes that Laspect, activated after iteration 500, cannot retroactively arrest.The warmup prevents spikes before the soft screen-space penalty becomes active.Source: §B.6 · p. 23
  • Removing the log-clamp permits accumulated log-scale drift beyond the inter-iteration budget, which Laspect cannot fully reabsorb as a soft penalty.The clamp limits scale drift after iteration 500.Source: §B.6 · p. 23
  • Removing Laspect allows gaussians to regrow into thin screen-space streaks when projection directions shift away from training views, defeating warmup-then-clamp alone.The anisotropy penalty protects against view-dependent streak formation after projection changes.Source: §B.6 · p. 23

C Limitations and Future Work

The FLAME-mesh foundation leaves large accessories under-modeled and depends on successful frontal face detection, while per-subject refinement still limits true real-time creation to the feed-forward stage.

Limitations

  • Large accessories such as glasses or hats are not explicitly modeled and may be folded into the head’s surface texture.This limitation is inherited from the FLAME-mesh foundation.Source: C Limitations and Future Work · p. 24
  • The upstream FLAME tracker assumes successful frontal face detection on at least one source frame.This assumption is another limitation inherited from the FLAME-mesh foundation.Source: C Limitations and Future Work · p. 24

Limitations and Future Work

  • ∼2 minutes: the per-subject refinement loop remains too slow for true real-time avatar creation, restricting real-time use to the feed-forward stage.The passage identifies this runtime as a limitation and motivates future work.Source: C Limitations and Future Work · p. 24

Related papers

Loading 2606.15659v1…