Source-linked AI summary
GHARP: Real-time Gaussian Head Animation from Large-scale Reconstruction Prior
Ali Benlalah, Sepehr Johari, Patricia Vitoria, Armin Kappeler, Artem Sevastopolsky, Alexander Jung, Gabriele Fanelli, Kevin Mader, Manuel Breitenstein, Claudia Plüss, Jan Rüegg, Simon Biland, Thomas Etterlin, Dmitry Kostiaev, Mathias Deschler, Brian Amberg, Sebastian Martin
TL;DR
Expression-driven head avatars face a scaling challenge, and facial expression codes leave body pose and clothing underspecified. GHARP separates offline identity construction from runtime residual animation and aligns body supervision; it reports state-of-the-art quality with substantially faster runtime and fewer Gaussians.
Problem
Per-subject Gaussian fitting requires dense captures and hours of optimization, while feed-forward pixel-aligned methods scale Gaussian count and inference cost with input views; expression codes also omit independent body and clothing variation.
Source: §1 · pp. 2–3Method
GHARP fuses input views into a canonical Gaussian representation and deformation latent, predicts expression-dependent residuals, and uses BAN to composite target-expression faces with reference-image bodies.
Source: §3 · p. 5, §3.1 · p. 6, §3.5 · p. 8Results
GHARP reports state-of-the-art quality, up to 13× faster A100 runtime than FastGHA and 86× faster than Avat3r, with 8× fewer Gaussians.
Source: §1 · p. 3, §5 · p. 14Takeaways & Limitations
The fixed-size Gaussian representation decouples animation cost from input-view count, while BAN improves temporal stability by resolving body-appearance inconsistencies during training.
Source: §1 · p. 3, §5 · p. 14
Abstract
from arXiv · showhide
We present GHARP (Real-time Gaussian Head Animation from Large-scale Reconstruction Prior), a method that animates 3D human heads in real time from a few input images of a subject and a driving expression signal. We decouple the problem into an identity stage that builds a representation of the subject's geometry and appearance offline, and an animation stage that predicts expression-dependent residuals on top of it at runtime. This separation offers a favorable trade-off with respect to fidelity, quality and runtime: the identity stage can be expensive while the animation stage runs a lightweight network, optimized for mobile devices. Our method performs animation in a semantically structured latent space of a pretrained reconstruction model, where expression changes remain spatially contained, making residual prediction efficient. This reconstruction prior provides a consistent spatial layout, allowing fusion of multiple input views into a compact, fixed-size canonical Gaussian representation. While this two-stage design improves the runtime-quality trade-off, it still inherits a problem common to all expression-driven avatar methods: expression codes describe only the face and thus omit body pose and clothing position, making these regions underspecified in the input. The animation network faces an ill-posed mapping and resorts to averaging over conflicting body appearances, producing blur and temporal flicker. We address this with a body alignment network that learns to align the person's body in the target image with the input reference images, removing the ambiguity from the training signal. Our method achieves state-of-the-art quality on the Ava-256 benchmark while running up to 13x faster on an A100 GPU with 8x fewer Gaussians.
1 Introduction
GHARP separates subject identity construction from runtime expression animation in a structured latent space, and uses body alignment to address body-appearance ambiguity. Its fixed-size representation supports constant-cost animation across input-view counts.
1 Introduction
- Per-subject Gaussian fitting requires dense captures and hours of optimization, while feed-forward pixel-aligned methods’ Gaussian count and inference cost grow linearly with input views.Source: §1 · p. 2
- A pretrained reconstruction prior provides semantically aligned latents, letting GHARP model expression changes as localized residuals on a canonical Gaussian map.The fixed Gaussian count decouples animation cost from reconstruction complexity and input-view count.Source: §1 · p. 2
- Multiple input views are fused into a canonical representation of facial geometry and appearance, alongside a deformation latent for subject-specific facial motion.Both representations are precomputed, leaving a lightweight convolutional network to predict runtime residuals from identity, deformation, and expression inputs.Source: §1 · pp. 2–3
- Body pose and clothing can vary independently of facial expression, so expression-only supervision creates conflicting body appearances and temporal jitter; BAN composites target-expression faces with reference-image bodies.Source: §1 · p. 3
- On an A100, GHARP is up to 13× faster than FastGHA and 86× faster than Avat3r, with 8× fewer Gaussians.Source: §1 · p. 3
2 Related Work
Prior head-avatar work spans optimized 3D representations, 2D reenactment, and feed-forward 3D methods, with trade-offs in fidelity, speed, viewpoint control, and scalability. GHARP uses a fixed-size identity representation and body alignment within a feed-forward architecture.
2 Related Work
- Earlier 3DMM methods [19] have limited capacity for details such as hair and clothing, while NeRF approaches often require costly scene optimization and render below real time.Source: §2 · p. 3
- Gaussian-splatting avatars [1] offer real-time rendering and high fidelity, but per-subject optimization limits generalization to unseen identities.GHARP instead uses a feed-forward architecture inspired by.Source: §2 · p. 4
- 2D reenactment methods [63] [75] provide fast, generalizable animation but lack true viewpoint control and explicit 3D geometry.Source: §2 · p. 4
- Feed-forward 3D methods [6] support sparse-input reconstruction, but pixel-aligned Gaussian approaches scale rendering cost and memory linearly with input views.GHARP contrasts this with a fixed-size identity representation and body alignment for consistency across views and time.Source: §2 · p. 4
- Some generalized accelerated avatars use linear blend skinning, which limits facial-expression rigging to linear motion of a coarse parametric model.GHARP’s architecture aggregates per-view latents into fixed-size canonical and deformation representations for runtime animation.Source: Figure 2 · p. 4, §2 · p. 5
3 Method
GHARP separates a subject’s fixed identity representation from runtime expression-driven residual animation. It builds that representation from input views in a pretrained reconstruction model’s UV-aligned latent space and uses body alignment to provide consistent animation supervision.
3 Method
- GHARP animates a canonical Gaussian UV map by adding expression-dependent residuals, keeping static appearance and geometry in C while D encodes facial deformation.The expression code w and D drive a lightweight animation network E; residual changes target non-rigid facial regions while largely preserving skin texture and pores.Source: §3 · p. 5
3.1 Offline Identity Subspace Learning
- The offline identity subspace combines input-view appearance and geometry in C and encodes subject-specific facial motion across expressions in D.Latents are fused with camera geometry into C, while a deformation encoder aggregates decoded input maps and C; detaching C from that encoder’s computation graph keeps the canonical path focused on expression-agnostic identity.Source: §3.1 · pp. 6–7, Figure 3 · p. 6
3.2 Animation Network
- The animation network E maps expression code w and deformation latent D to a residual for each Gaussian parameter in the canonical UV map.A ResNet decoder processes the spatial UV layout for efficient GPU execution.Source: §3.2 · p. 7
3.3 Passthrough Auxiliary Loss
- A passthrough auxiliary loss anchors the canonical map to input views by rendering corrected Gaussian maps and supervising them against the matching images.The narrow-bottleneck corrections affect only Gaussian positions and opacities, encouraging C to retain appearance attributes rather than relying on per-view reconstruction.Source: §3.3 · p. 7
3.4 Training Losses
- Training combines image reconstruction, perceptual, regularization, eye-and-mouth region, and passthrough losses.The total loss includes L1, SSIM, LPIPS, region, regularization, and passthrough terms.Source: §3.4 · pp. 7–8
3.5 Body Alignment Network (BAN)
- BAN addresses body-pose and clothing variation that expression codes do not explain by compositing a target face with body appearance from a reference image.It produces aligned supervision targets in latent space; replacing raw targets with their renderings supplies a consistent signal during animation-network training.Source: §3.5 · pp. 8–9
4 Experiments
Experiments on Ava-256 and Internal10K evaluate image quality, expression transfer, identity preservation, body stability, and runtime. GHARP achieves strong benchmark results with fewer Gaussians, while BAN improves body stability and design ablations identify useful components.
4.1 Datasets and Expression Codes
- Internal10K training uses 100 expression-diverse frames per subject across 10K identities; evaluation uses 100 frames from 50 held-out subjects, while Ava-256 evaluation uses 12 held-out subjects.Expression codes are 51 ARKit blendshape weights; Ava-256 training uses 244 subjects with approximately 1,000 frames each.Source: §4.1 · p. 9
4.2 Experimental Setup
- The two-stage training freezes pretrained HeadsUp components before training remaining modules, then fine-tunes on Ava-256; evaluation includes image, face-specific, body-stability, and retargeting metrics.Baselines were reimplemented because their code was unavailable.Source: §4.2 · pp. 9–10
4.3 Main Results
- Ours and Ours+BAN achieve state-of-the-art Ava-256 results; GHARP leads AKD and CSIM across datasets and input counts, uses 8× fewer Gaussians, and runs up to 13× faster on A100 and 31× on iPhone 17 Pro ANE.LPIPS and SSIM remain close to the best baseline; BAN's pixel-aligned metrics can degrade because its body aligns to a reference image rather than ground truth.Source: Table 1 · p. 10, §4.3 · p. 10, §4.4 · p. 11
4.4 Analysis and Ablation Study
- Removing the canonical representation or deformation latents worsens metrics or identity detail, while the passthrough loss improves texture preservation and geometric consistency.BAN's pixel-aligned metric drop reflects body alignment to a reference rather than compromised face quality; regional results show face quality remains essentially unchanged.Source: §4.4 · pp. 11, 13, Table 4 · p. 13
4.3 Main Results
- Ours+BAN reduces expression-independent body jitter and produces a sharper body; the temporal-stability comparison shows jitter for Ours and the baselines but a stable body with BAN.Qualitative comparisons also report sharper facial details and more accurate expression transfer for GHARP.Source: §4.3 · pp. 10–11, Figure 5 · p. 12, Figure 6 · p. 12
4.4 Analysis and Ablation Study
- Runtime is constant across input counts for GHARP, whereas FastGHA scales linearly; cross-identity retargeting preserves expression detail better than the compared methods by AED.The cross-identity comparison reports shoulder artifacts for Avat3r and loss of fine details for FastGHA.Source: §4.4 · pp. 13–14, Figure 9 · p. 14
5 Conclusion
GHARP uses a learned canonical Gaussian representation to decouple animation cost from input-view count. BAN addresses body appearance inconsistencies during training, and the method's stated limitation is dependence on calibrated camera poses.
5 Conclusion
- GHARP animates heads on a learned canonical Gaussian representation, capturing subject appearance and geometry in a fixed-size set that decouples animation cost from input-view count.The conclusion reports state-of-the-art quality with up to 8× fewer Gaussians and an order of magnitude lower inference cost.Source: §5 · p. 14
- BAN addresses body-region appearance inconsistencies during training and improves temporal stability at animation time.Source: §5 · p. 14
- The method depends on calibrated camera poses, so inaccurate pose estimates could propagate through the encoder.The conclusion identifies robustness to temporal input variations or noise and material factorization for portrait relighting as possible extensions.Source: §5 · p. 14
A.1 Network Architecture
The appendix describes convolutional networks for fusing input-view latents, encoding deformation, predicting animation residuals, and adding passthrough corrections. The resulting Gaussian maps are activated into the final representation.
A.1 Network Architecture
- FC fuses per-view latents and Plücker embeddings into spatial blending weights and a correction term at constant 32×32 resolution.Its final convolution is zero-initialized so it initially outputs the unweighted mean of the input latents.Source: §A.1 · p. 21, Table 5 · p. 21
- FD processes each input Gaussian UV map and the canonical map independently, then consolidates their features into a deformation latent.The per-map U-Net produces 64×64 features; the consolidation network also operates at 64×64.Source: §A.1 · p. 21, Table 6 · p. 22, Table 7 · p. 22
- E combines the deformation latent with the expression code and predicts an expression-dependent residual map for the Gaussian UV representation.The expression code is expanded by an MLP, spatially tiled, and fused with the deformation latent before residual blocks and upsampling.Source: §A.1 · p. 21, Table 8 · p. 23
- The residual is added in pre-activation parameter space, with scaling that initially passes through canonical-frame details and improves detail preservation and convergence speed.Per-attribute activations then produce the final Gaussian UV map.Source: §A.1 · pp. 22–23
- The passthrough network predicts corrections on top of the canonical representation by combining a compressed input latent with detached deformation features.Its encoder-decoder outputs a correction at Gaussian UV resolution.Source: §A.1 · p. 23
A.2 Training Setup
The appendix reports shared training and validation splits for pretraining and reimplemented baselines, notes that the dataset is not public, and provides a passthrough-network architecture table.
A.2 Training Setup
- Pretraining samples 100 expression-diverse frames per subject across 10K identities and evaluates on 100 frames from 50 held-out subjects; the frozen prior, new modules, and baselines share train/validation splits.Source: §A.2 · p. 23
- The appendix includes a table describing the passthrough-network architecture.Source: Table 9 · p. 24
- The dataset is not publicly available, and participants gave written consent for data use; subjects shown in the paper gave additional consent for image publication.Source: §A.2 · p. 24
A.3 Training Losses
The training objective combines image reconstruction, perceptual, Gaussian-regularization, and region-based losses. Landmark information from the target frame also contributes to the loss.
A.3 Training Losses
- The loss also uses 2D facial landmarks detected in the target frame.Source: §A.3 · pp. 24–25
- The total objective combines L1, SSIM, LPIPS, region-based, Gaussian-regularization, and passthrough losses.The objective is written as L = λ1L1 + λssimLssim + λlpipsLlpips + λregionLregion + λregLreg + Lpass.Source: §A.3 · p. 24
- L1 and SSIM losses compare rendered images with ground truth, while LPIPS measures perceptual differences using pretrained AlexNet features.The losses are L1 = ∥Ĩ − IGT∥1 and 1 − SSIM(Ĩ, IGT).Source: §A.3 · p. 24
- Opacity and scale regularization encourage Gaussians to remain opaque and within reasonable scales.The scale penalty is zero for scales from 0.1 through 10 and penalizes values outside that range.Source: §A.3 · p. 25
A.4 Baselines
The evaluation compares GHARP with feed-forward head-animation baselines under a shared pretraining and finetuning protocol, alongside a separately adapted UIKA comparison on Ava-256. The baseline implementations include changes intended to support training or improve their performance.
A.4 Baselines
- Avat3r and FastGHA are pretrained on Internal10K and finetuned on Ava-256 using the same two-stage protocol as GHARP.On Ava-256, the comparison also includes concurrent UIKA.Source: §A.4 · p. 26
- The Avat3r reimplementation replaces costly DUSt3R extraction with DINOv3, adds expression conditioning, and includes DINOv2 loss and additional regularizations.The authors report that DINOv2 loss and the regularizations improve performance compared with a literal reimplementation.Source: §A.4 · p. 26, Figure 10 · p. 28
- FastGHA is reimplemented with added learnable encoders and a residual decoder, plus staged training, silhouette loss, and scale penalties.The authors report that these changes only marginally improve quality and were added to strengthen the baseline.Source: §A.4 · pp. 26–27
- UIKA is finetuned on Ava-256 with its architecture and core training components unchanged, while an MLP maps ARKit codes to FLAME parameters.The comparison uses the same held-out subjects and four test input views as GHARP; UIKA renders 134K Gaussians.Source: §A.4 · pp. 27–28
A.5 Body Alignment Network (BAN)
BAN learns to blend reference and target latents to create aligned supervision, using separate face and body targets and masked reconstruction losses. Its training also applies an adversarial loss to full renderings.
A.5 Body Alignment Network (BAN)
- A six-layer convolutional network blends concatenated reference and ground-truth latents at 32×32 resolution, outputting weights and a correction tensor.Zero-initializing the final convolution makes the initial blend a 50/50 average.Source: §A.5 · p. 28
- BAN supervises the aligned rendering's face against the target and its body against the reference, using masked L1 and LPIPS losses plus a full-rendering adversarial loss.Face and body masks come from pretrained parsing and segmentation networks; randomized mask boundaries help prevent seams.Source: §A.5 · p. 29, Table 11 · p. 29
- BAN is trained for 20K iterations with Adam at a learning rate of 5×10−5 and batch size 16.Source: §A.5 · p. 29
A.6 Body Temporal Stability Analysis
Across the supplementary analyses, BAN is the most stable across three body-temporal metrics, while frozen-body blending trades seam visibility against chin shape and expression fidelity. GHARP's animation runtime remains constant across input-view counts, and the additional evaluations examine input-view scaling, prior strength, identity latency, in-the-wild use, retargeting, and UIKA.
A.7 Stable vs. Frozen Body
- A frozen body yields σ_t body = 0, but blending it into the animated map either leaves a neck seam or alters chin shape and reduces expression fidelity.Ours+BAN instead gives σ_t body = 0.7 and retains realistic neck motion; BAN avoids a blending boundary by predicting a consistent Gaussian UV map.Source: §A.7 · pp. 29–30, Figure 11 · p. 30
A.6 Body Temporal Stability Analysis
- Ours+BAN is the most stable across RGB temporal deviation, CIELAB color-flicker deviation, and optical-flow drift.The latter two metrics measure perceptual color flicker and geometric drift, respectively.Source: §A.6 · p. 29, §A.7 · p. 30, Table 12 · p. 30
A.8 Runtime Performance
- At four input views, GHARP is 7.7× faster than FastGHA on H100 and 13× faster on iPhone 17 Pro GPU; end-to-end A100 latency is 3.25 ms.Animation time remains constant as input-view count changes, unlike Avat3r and FastGHA; with one view, GHARP is 2× faster on H100 and 3.5× faster on iPhone 17 Pro GPU. Reconstruction quality scales with input images, and even at N = 1 GHARP is described as sharper and better at preserving identity than prior work.Source: §A.8 · pp. 30–31, Table 13 · p. 31, Table 15 · p. 32
A.10 Prior Contribution
- With a HeadsUp prior trained on 1k rather than 10k subjects, reconstruction quality decreases but remains comparable on SSIM and CSIM to Avat3r and FastGHA.Ours1k achieves AKD 7.8, compared with 8.3 for Avat3r and 9.3 for FastGHA.Source: §A.10 · p. 32, Table 16 · p. 32
A.11 Identity Network Performance
- The identity network runs in 71 ms, compared with 242 ms for FastGHA and 2384 ms for Avat3r, on one A100 with four input views.GHARP runs this network only once per identity; reducing Avat3r's Sapiens resolution still leaves its measured latency above GHARP's.Source: §A.11 · p. 33
A.12 In-The-Wild Setting
- In an iPhone-recorded self-reenactment, GHARP preserves more fine detail and identity from one input image than FastGHA and Avat3r.The method uses ARKit expression codes and a HeadsUp prior trained on studio data; adapting the prior to more diverse capture settings is left for future work.Source: §A.12 · p. 33
A.13 Cross-Identity Retargeting
- In cross-identity retargeting, GHARP shows stronger expression transfer than Avat3r and FastGHA, including more extreme expressions and expression-dependent wrinkles.Source: §A.13 · p. 33, Figure 14 · p. 35
A.14 Comparison with UIKA
- Against UIKA on Ava-256, GHARP achieves the best AKD and PSNR with half as many Gaussians, while UIKA is slightly better on LPIPS and SSIM.GHARP's animation stage is faster on every device where both run; UIKA cannot run on the iPhone ANE because of its large LBS matrix multiplications.Source: §A.14 · p. 34, Table 17 · p. 35