PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialAvatar-0: High-Quality 4D Head Avatar with Multi-Stage Reconstruction

Yiran Wang, Zeyu Zhang, Yuanming Li, Ziming Wang, Yang Zhao

arXiv:2606.15659v1cs.CV

TL;DR

Existing 4D head-avatar systems separate feed-forward prediction from costly per-subject refinement, limiting their ability to share a common representation. SpatialAvatar-0 unifies both regimes with a shared FLAME-mesh-bound Gaussian pipeline, achieving stronger cross-domain and per-subject results with substantially shorter refinement.

  • Problem

    Existing feed-forward predictors use fixed source counts and single-dataset training, while per-subject refiners are costly and disrupt Gaussian layouts, hindering a shared representation.

  • Method

    SpatialAvatar-0 uses a shared FLAME-mesh-bound Gaussian representation with variable-source feed-forward generation, two-phase training, and 10K-iteration layout-preserving refinement.

  • Results

    +1.5 dB PSNR over GAGAvatar on zero-shot VFHQ/HDTF and +1.3 dB PSNR over 300K-iteration GeoAvatar while leading every reported SplattingAvatar metric.

  • Takeaways & Limitations

    The results demonstrate that generalizable feed-forward and per-subject 3DGS avatar regimes can be developed within one coherent pipeline.

  • Takeaways & Limitations

    Per-subject refinement remains approximately two minutes, so true real-time avatar creation is limited to the feed-forward stage.

Abstract

from arXiv · show

High-quality 4D head avatars from one or a few source portraits are central to telepresence, AR/VR, and digital-human interaction. 3D Gaussian Splatting (3DGS) has emerged as the dominant representation, with two complementary regimes (generalizable feed-forward predictors and per-subject refiners) maturing in parallel. However, existing feed-forward predictors are trained on a single dataset family with a hard-coded source count, inheriting the corresponding domain bias. Per-subject refiners require 300K--600K iterations and rely on adaptive densification that destroys upstream Gaussian layouts, preventing the two regimes from sharing a representation end-to-end. To bridge both regimes we propose SpatialAvatar-0 on a shared FLAME-mesh-bound Gaussian representation: a feed-forward generator with a parameter-free K-source mean-pool and a monocular-temporal to multi-view-spatial two-phase schedule that anchors against identity-prior collapse onto the smaller multi-view set. We further introduce a 10K-iter layout-preserving per-subject refinement loop that freezes the FLAME-binding and Gaussian count and replaces densification with a three-component anti-spike regularization. On VFHQ/HDTF cross-domain zero-shot we surpass the in-domain leader GAGAvatar by +1.5 dB PSNR despite never training on either test domain, and on the SplattingAvatar monocular benchmark we lead every reported metric, surpassing the 300K-iter GeoAvatar by +1.3 dB PSNR at up to 60x shorter per-subject schedule than common SOTA baselines. Website: https://spatialwalk.github.io/SpatialAvatar-0.

1 Introduction

SpatialAvatar-0 unifies feed-forward prediction and per-subject refinement through a shared FLAME-mesh-bound Gaussian representation. It addresses dataset and source-count constraints in feed-forward models while enabling faster, layout-preserving refinement without densification.

  • Motivation: 4D head avatars support telepresence, AR/VR communication, and digital-human interaction, motivating both generalizable feed-forward predictors and per-subject 3DGS refiners.Feed-forward predictors generate animatable face-bound Gaussian models from one or a few portraits, while refiners optimize individual subjects.
  • Problem: Existing feed-forward avatars inherit single-dataset domain bias and hard-code source-image count, limiting multiview geometry and variable-source deployment.Monocular training lacks ground-truth multiview geometry, while fixed source counts prevent exploiting arbitrary numbers of available views.
  • Problem: 300K–600K iterations per identity and adaptive densification make per-subject refinement slow and destroy feed-forward spatial inductive bias, while removing densification causes anisotropic Gaussian overfitting.Without densification, Gaussians can elongate along one axis and catastrophically break novel-view rendering.
  • Method: SpatialAvatar-0 uses a shared FLAME-mesh-bound Gaussian representation, a parameter-free K-source mean-pool, and monocular-temporal →multi-view-spatial training to combine identity coverage with geometric grounding.The schedule uses L2-SP anchoring and a 25% NeRSemble cross-time mix to prevent identity-prior collapse onto the smaller multi-view set.
  • Method: 10K-iter layout-preserving refinement replaces densification with three-component anti-spike regularization and leads every reported SplattingAvatar metric at up to 60× shorter schedules.The refinement loop freezes the inherited Gaussian layout while avoiding the long schedules of common per-subject baselines.

2 Related Work

Related work progresses from 2D talking-head synthesis toward 3D-aware representations, culminating in 3D Gaussian Splatting for efficient, geometrically faithful head avatars. SpatialAvatar-0 uses a hybrid representation with multi-scale fusion and enhanced rigging to support view-consistent rendering without per-identity optimization.

  • Overview: Head-avatar research targets photorealistic, animatable reconstruction from sparse inputs while controlling facial expressions, gaze, and head poses.Applications include telepresence, gaming, and the metaverse.
  • 2D Methods: Early talking-head methods used 2D generative models, feature modulation, latent manipulation, and dense warps to synthesize or deform portrait sequences.3D Morphable Models were added as low-dimensional geometric priors to improve deformation realism and structural integrity.
  • 3D-Aware Representations: 3D-aware approaches emerged to address view consistency, progressing from mesh-based statistical face priors to flexible, topology-agnostic NeRF reconstruction.These methods reconstruct personalized avatars from monocular or sparse inputs.
  • 3D Gaussian Splatting: 3D Gaussian Splatting balances rendering efficiency and geometric fidelity through explicit anisotropic primitives and has been extended to head modeling with mesh or deformation-field bindings.Its tile-based rasterization enables real-time rendering.
  • SpatialAvatar-0: In contrast, SpatialAvatar-0 combines multi-scale feature fusion with enhanced rigging for high-fidelity, view-consistent rendering without time-consuming per-identity optimization.The hybrid representation bypasses redundant volumetric-rendering computations while maintaining generalization capabilities.

3 Method

SpatialAvatar-0 reconstructs heads in two coupled stages over a shared FLAME-mesh-bound Gaussian representation: variable-source feed-forward generation followed by optional 10K-iteration per-subject refinement. Its method combines two-phase monocular-to-multi-view training with fixed UV-pixel Gaussian layouts, frozen bindings, and anti-spike regularization.

  • Two-stage reconstruction: The pipeline uses a shared FLAME-mesh-bound 3D Gaussian representation with a feed-forward generator and optional per-subject photometric refinement.The generator accepts K ∈ {1, 2, 3, 4} source portraits, while refinement starts from its output for a chosen reference frame.
  • Two-phase training: Training proceeds from monocular-temporal CelebV-HQ pretraining to multi-view-spatial NeRSemble post-training without architectural changes.Both phases expose the model to variable K ∈ {1, 2, 3, 4} source contexts; Phase 2 samples synchronized cameras across views and time.
  • Gaussian representation: Each identity contains approximately 58K Gaussians, one per valid pixel on a 256×256 FLAME UV grid, rigidly bound to parent triangles.The UV-pixel organization is fixed infrastructure, and the binding is preserved during refinement.
  • Architecture: The architecture combines frozen DINOv3 features, UV-space StyleUNet generation, target-camera harmonic encoding, and FLAME-conditioned residual corrections.The residual head predicts per-attribute corrections for position, rotation, scale, and color, while shape β enters through FLAME geometry.
  • Per-subject refinement: The 10K-iteration refinement freezes the FLAME mesh and triangle bindings, preserving the one-to-one UV-pixel-to-Gaussian map while optimizing against randomly sampled target frames.The mesh is deformed to each target pose, transformed Gaussians are rendered, and photometric loss is optimized.
  • Per-subject refinement: Anti-spike regularization combines a 500-iteration scale-freeze warmup, a hard log-scale clamp, and a soft screen-space anti-anisotropy penalty.These mechanisms address Gaussian elongation along a single axis when densification is removed.

4 Experiments

Experiments evaluate SpatialAvatar-0 in feed-forward zero-shot and per-subject settings across out-of-distribution and monocular benchmarks, with ablations on held-out CelebV-HQ. The method achieves leading image-quality results while substantially reducing per-subject training cost.

  • Evaluation setup: The evaluation covers VFHQ and HDTF for feed-forward zero-shot reenactment, SplattingAvatar for per-subject evaluation, and held-out CelebV-HQ for ablations.Metrics include PSNR, SSIM, LPIPS, and feed-forward identity, expression, pose, and keypoint measures.
  • Feed-forward comparison: Ours-FF leads every reported column on VFHQ and HDTF, surpassing the in-domain leader GAGAvatar by +1.5 dB PSNR on each split.The test domains are out-of-distribution for SpatialAvatar-0, whereas several baselines include one or both datasets during training.
  • Feed-forward comparison: Ours-FF preserves high-frequency identity cues, including forehead wrinkles, malar elevation, beard texture, nasolabial folds, and hair-edge silhouette across identities and poses.These qualitative findings complement the quantitative gains across the reported feed-forward metrics.
  • Per-subject comparison: Ours+S3 leads every reported SplattingAvatar metric and surpasses GeoAvatar by +1.3 dB PSNR at 10K versus 300K iterations.The comparison covers MSE, PSNR, SSIM, and LPIPS, with GeoAvatar previously leading PSNR, SSIM, and MSE.
  • Per-subject efficiency: ∼2 min per subject is two orders of magnitude shorter than FlashAvatar, GeoAvatar, and GaussianAvatars on a single RTX 3090.The 10K-iteration schedule is 15×, 30×, and 60× shorter than those baselines, respectively.
  • Pipeline and architecture ablation: The headline ablation uses self-reenactment on a held-out CelebV-HQ slice, cumulatively builds the pipeline, and removes architectural components from the full configuration.Adding Phase-2 NeRSemble v2 post-training without controls catastrophically forgets CelebV-HQ; the L2-SP anchor and 25% cross-time mix are then reintroduced.

5 Conclusion · A Additional method details

SpatialAvatar-0 bridges feed-forward and per-subject 3DGS head-avatar regimes with a shared FLAME-mesh-bound Gaussian representation. Its variable-source generator, two-phase schedule, and 10K-iteration layout-preserving refinement target cross-domain identity priors and efficient personalization.

  • 5 Conclusion: SpatialAvatar-0 unifies generalizable feed-forward and per-subject 3DGS head-avatar regimes through a shared FLAME-mesh-bound Gaussian representation.
  • 5 Conclusion: The feed-forward generator uses a K-source-variable design and a monocular →multi-view two-phase schedule.
  • 5 Conclusion: +1.5 dB PSNR is achieved on zero-shot VFHQ/HDTF benchmarks over the in-domain leader.
  • 5 Conclusion: The two-phase generator produces a strong cross-domain identity prior for zero-shot evaluation.
  • 5 Conclusion: Per-subject personalization uses a 10K-iter layout-preserving refinement procedure.
  • 5 Conclusion: The refinement procedure is enabled by mechanisms described in the continuation of the conclusion passage.
  • A Additional method details: Appendix A collects derivations, exact functional forms, and training-recipe specifics referenced from Section 3.
  • A Additional method details: The additional method details section provides supporting mathematical and training documentation for the main method.

A.1 Feed-forward representation and architecture

SpatialAvatar-0 uses a topology-respecting UV-aligned Gaussian representation whose mesh-bound structure supports convolutional generation with content-independent Gaussian count. Parameter-free barycentric warping, per-triangle local frames, harmonic view encoding, and expression-conditioned residual FiLM heads provide the inputs and attribute refinements for feed-forward prediction.

  • UV-aligned representation: The UV-aligned representation maps neighboring UV pixels to neighboring mesh triangles, transfers UV smoothness to the mesh surface, and fixes Gaussian count independently of image content.Its topology-respecting unwrap gives 2D convolutions an inductive bias for face-bound primitives.
  • Barycentric feature warp: A parameter-free barycentric warp identifies each UV pixel’s FLAME triangle, interpolates its source-frame 3D position, projects it through the source camera, and samples encoder features and RGB.The warp uses precomputed triangle assignments, barycentric weights, and bilinear sampling.
  • Harmonic view-direction encoding: The target camera direction is unit-normalized and represented with a 27-dimensional harmonic embedding for view-dependent conditioning.The direction is taken from the third column of the target view’s world-to-camera matrix.
  • Residual head conditioning: A 2-layer MLP generates per-attribute FiLM coefficients for position, rotation, scale, and color, whose 1 × 1 convolutional outputs refine the decoded UV maps while excluding opacity.The residual head conditions on target expression, pose, and eye codes; shape is excluded because it is fixed per subject and directly feeds mesh geometry.
  • Per-triangle frame construction: Per-triangle local frames use the centroid for translation, Gram-Schmidt-like edge-and-normal orthonormalization for rotation, and averaged edge-length and altitude geometry for isotropic scale.These frames define the local basis for Gaussian attributes and quaternion corrections.

A.2 Training procedure

Training uses difficulty-weighted source sampling and a two-phase schedule that combines near-uniform early exploration with predominantly cross-camera multi-view supervision in Phase 2. The procedure specifies explicit loss handling and distinct optimization settings across phases.

  • Source sampling: Source frames are sampled using a softmax over camera-direction and expression-code dissimilarity to avoid trivial near-identity source-target pairs.The softmax temperature is annealed from high, approximately uniform sampling toward low-temperature focused sampling during early training.
  • Phase 2 sampling: Phase 2 samples K+1 cameras from synchronized 16-camera NeRSemble captures, using 75% cross-camera batches and 25% cross-time batches by default.Cross-camera draws provide multi-view supervision, while cross-time draws preserve temporal variation; when K=1, the latter reduce to monocular same-camera-different-time sampling.
  • Loss formulation: All UV-domain loss terms are averaged over the valid-UV mask, while TV penalties operate on the UV grid and can smooth across FLAME UV seams.The valid mask contains approximately 58,173 pixels, and the cross-seam contribution is described as small in this setting.
  • Optimization: Phase 1 uses Adam at 2.5×10−4 with FP32, batch size 48, and gradient clipping at norm 5.0; Phase 2 uses AdamW with cosine warmup, bf16, and batch size 12.Phase 2 additionally sets λsp=10−3, with a 25% NeRSemble cross-time mix share; Lδ is inactive in Phase 1.

A.3 Jacobian penalty derivation … B.3 Backbone capacity and source-image count

The paper derives an efficient Hutchinson-based Jacobian regularizer for view-direction invariance, specifies refinement safeguards and initialization details, and reports qualitative evidence across monocular, cross-identity, backbone, and source-count settings. The ablations show sharper reconstructions, monotonic backbone gains, and diminishing returns beyond two source images.

  • A.3 Jacobian penalty derivation: The regularizer drives position and scale Jacobians with respect to view encoding γ(d) toward zero at every UV pixel, while leaving color and opacity view-dependent.Rotation is excluded because raw quaternion-component penalties do not represent angular-Jacobian energy under the unit-sphere constraint.
  • A.3 Jacobian penalty derivation: A Hutchinson per-pixel estimator avoids the one-to-two-orders-of-magnitude autograd overhead of explicit per-pixel Jacobian evaluation.The finite-difference alternative is rejected because moderate-to-large angular separations confound first-order sensitivity with higher-order curvature.
  • A.3 Jacobian penalty derivation: The log(1 + ·) wrapper caps early-training regularizer magnitude, when the unwrapped argument is approximately 10^2, while approaching identity as the argument shrinks.The wrapped objective incurs a second-order Jensen bias that decreases as the expected Jacobian energy approaches zero.
  • A.4 Per-subject optimization details: Per-subject refinement combines a hard parameter-space log-scale clamp with a soft screen-space projected-aspect penalty, enforcing world-space anisotropy bounds while encouraging isotropic footprints.The two controls act on different quantities and are therefore not redundant.
  • A.4 Per-subject optimization details: Reference-frame initialization selects the minimum jaw-pose magnitude, but falls back to a random bottom-decile frame when the minimum is below τjaw = 10^-3.The heuristic is jaw-only and does not guarantee a fully canonical pose.
  • B.1 Per-subject monocular qualitative comparison: Ours+S3 follows held-out target poses closely without visible blur or dropped regions, preserving wrinkles, facial contours, hair edges, and eye highlights at rendered scale.The visualization sample contains four subjects and compares Ours+S3 with five per-subject baselines.
  • B.2 Cross-identity feed-forward qualitative comparison: Ours-FF retargets source identity under substantially different driver yaw and expression while preserving source-consistent eye contours, nose-bridge orientation, and jaw outline.The comparison covers VFHQ and HDTF test splits against GAGAvatar, CVTHead, GPAvatar, and Portrait4D-v2.
  • B.3 Backbone capacity and source-image count: DINOv3 backbone quality improves monotonically from S to L, with DINOv3-B near the quality–parameter knee, while source-count gains are largest from K=1 to K=2 and flatten thereafter.The encoder is frozen, and the source-count ablation evaluates K ∈ {1, 2, 3, 4} at inference time.

B.4 Training hyperparameter ablation

This ablation varies four feed-forward training hyperparameters while holding the others at default settings. LinearLR and a Phase 1 base learning rate near 2.5×10−4 are most influential, whereas Phase 2 layer-wise decay is comparatively insensitive on PSNR.

  • Ablation setup: The ablation varies the Phase 1 learning-rate schedule, Phase 1 base learning rate, Phase 2 encoder layer-wise LR decay γLR, and Jacobian penalty weight λjac independently.Row (a) reproduces Tab. 5 row (C), while Default rows (e), (h), and (k) are independent reruns.
  • Phase 1 schedule: ∼0.1–0.5 dB PSNR: LinearLR outperforms cosine and constant Phase 1 schedules, with constant LR performing worst.These comparisons correspond to rows (a)–(c).
  • Phase 1 base learning rate: 2.5×10−4: the Phase 1 base learning rate forms a single PSNR peak, while halving or doubling it loses ∼0.2–0.5 dB.The comparison uses rows (d)–(f).
  • Phase 2 layer-wise decay: Under 0.2 dB: deviations in either direction for Phase 2 layer-wise decay γLR regress PSNR, making it the least sensitive setting.The L2-SP anchor and cross-time mix provide the Phase 2 cross-domain protection.

B.5 L2-SP anchor and cross-time mix ablation

The ablation evaluates the L2-SP anchor and NeRSemble cross-time mixing in Phase 2, finding complementary cross-domain protection and sensitivity to anchor strength. The adopted setting is λsp=10−3 with 25% cross-time mix.

  • Ablation design: The 2×2 factorial isolates λsp ∈{0, 10−3} and cross-time mix share∈{0%, 25%} to measure each Phase 2 control individually and jointly.Rows (a)–(d) form the factorial design, while rows (e)–(f) vary λsp at the chosen mix share.
  • Complementarity: The anchor alone and cross-time mix alone each provide partial cross-domain protection, whereas their joint configuration recovers the full effect.This indicates the two mechanisms are complementary rather than redundant.
  • Anchor sensitivity: λsp=10−3 with 25% cross-time mix is adopted because under-anchoring approaches cross-time-mix-only performance, while over-anchoring limits multi-view signal absorption.The sensitivity comparison uses λsp∈{10−2, 10−4} at the chosen mix share.

B.6 Anti-spike component ablation

The anti-spike regularization combines a scale-freeze warmup, post-500 log-scale clamp, and screen-space anti-anisotropy penalty. Ablations show each component blocks a distinct failure mode that the remaining components cannot fully correct.

  • B.6 Anti-spike component ablation: Tab. 9 compares the full anti-spike configuration with variants removing the scale-freeze warmup, log-scale clamp, or Laspect while keeping the other two components fixed.The warmup spans iterations 0–500; the clamp and Laspect activate from iteration 500.
  • B.6 Anti-spike component ablation: Removing the scale-freeze warmup restores early anisotropy spikes that Laspect, activated after iteration 500, cannot retroactively arrest.The warmup prevents spikes before the soft screen-space penalty becomes active.
  • B.6 Anti-spike component ablation: Removing the log-clamp permits accumulated log-scale drift beyond the inter-iteration budget, which Laspect cannot fully reabsorb as a soft penalty.The clamp limits scale drift after iteration 500.
  • B.6 Anti-spike component ablation: Removing Laspect allows gaussians to regrow into thin screen-space streaks when projection directions shift away from training views, defeating warmup-then-clamp alone.The anisotropy penalty protects against view-dependent streak formation after projection changes.

C Limitations and Future Work

The FLAME-mesh foundation leaves large accessories under-modeled and depends on successful frontal face detection, while per-subject refinement still limits true real-time creation to the feed-forward stage.

  • Limitations: Large accessories such as glasses or hats are not explicitly modeled and may be folded into the head’s surface texture.This limitation is inherited from the FLAME-mesh foundation.
  • Limitations: The upstream FLAME tracker assumes successful frontal face detection on at least one source frame.This assumption is another limitation inherited from the FLAME-mesh foundation.
  • Limitations and Future Work: ∼2 minutes: the per-subject refinement loop remains too slow for true real-time avatar creation, restricting real-time use to the feed-forward stage.The passage identifies this runtime as a limitation and motivates future work.
Loading 2606.15659v1…