Source-linked AI summary
FFAvatar: Few-Shot, Feed-Forward, and Generalizable Avatar Reconstruction
Thuan Hoang Nguyen, Jiahao Luo, Yinyu Nie, Hao Li, Gordon Guocheng Qian, Jian Wang
TL;DR
Avatar reconstruction is hindered by hours-long per-subject optimization and large image requirements. FFAvatar reconstructs animatable 3D Gaussian head avatars from few-shot unposed portraits using a feed-forward multi-view framework, achieving broad benchmark improvements and real-time inference.
Problem
Existing avatar reconstruction methods require hours of per-subject optimization and dozens to hundreds of images, limiting rapid deployment from minimal subject-specific data.
Method
FFAvatar jointly fuses multiple unposed views into a canonical Gaussian avatar and predicts FLAME animation parameters directly from pixels through a multi-stage training strategy.
Results
+2.57 PSNR over GAGAvatar and +0.21 CSIM demonstrate improvements across NeRSemble rendering metrics, with four views further increasing performance.
Takeaways & Limitations
FFAvatar provides a scalable foundation for real-time, controllable human avatar synthesis, reconstructing avatars in under 2 seconds and animating at 49 FPS on one NVIDIA A100.
Takeaways & Limitations
FFAvatar is bounded by FLAME’s solution space and sparse views may miss fine details or boundaries, causing artifacts under extreme novel views.
Abstract
from arXiv · showhide
Avatar reconstruction has traditionally relied on per-subject optimization that requires hours of computation or on expensive preprocessing that limits scalability. We introduce FFAvatar, a generalizable feed-forward framework that reconstructs high-quality, animatable 3D Gaussian head avatars from few-shot unposed portrait images in seconds. FFAvatar fuses information from multiple source images into a unified canonical Gaussian representation through Multi-View Query-Former, which is animated via FLAME parameters predicted end-to-end directly from pixels, eliminating the overhead of offline FLAME extraction. We further propose a three-stage training curriculum that achieves both broad generalization and high-fidelity reconstruction: (i) scalable pretraining on extensive monocular video data with over 1M identities to learn strong generalizable priors; (ii) multi-view fine-tuning on a small but high-quality dataset of 360-degree captures to enhance geometric fidelity and extreme-view awareness; and (iii) optional personalization that adapts to specific identities for maximum fidelity within 500 optimization steps. Extensive experiments demonstrate that FFAvatar sets a new standard for identity preservation, geometric consistency, and animation fidelity. On the NeRSemble benchmark, it outperforms the state-of-the-art LAM by a substantial 5.5 PSNR gain. Furthermore, FFAvatar enables real-time deployment, reconstructing avatars in 2 seconds without personalization and 10 seconds with personalization, while supporting 49 FPS animation on a single NVIDIA A100 GPU.
1 Introduction
FFAvatar addresses the scalability and fidelity limits of prior avatar reconstruction by using multiple unposed portraits in a feed-forward framework for unseen identities. Its contributions combine a three-stage training curriculum, end-to-end FLAME estimation, and joint multi-view canonical reconstruction.
- Prior methods require hours of per-subject optimization and dozens to hundreds of images per identity, limiting rapid deployment with minimal subject-specific data.
- Single-view feed-forward reconstruction constrains identity preservation and geometric fidelity, especially for unseen or extreme viewpoints with occluded or poorly observed regions.
- FFAvatar reconstructs animatable 3D head avatars for unseen identities from multiple unposed portrait images in a single feed-forward pass.
- The framework predicts FLAME parameters directly from raw pixels through photometric supervision and jointly aggregates multiple views into one unified canonical Gaussian representation.
- Its three-stage curriculum combines scalable pretraining, multi-view fine-tuning, and optional personalization to achieve broad generalization and high-fidelity reconstruction.
2 Related Work
Prior avatar reconstruction methods achieve high-quality results through per-identity optimization but require hours to days, motivating feed-forward methods that reconstruct avatars in a single pass. Existing feed-forward approaches use parametric priors yet face limitations in photorealism, synthesis focus, or reliance on precomputed FLAME parameters.
- Optimization-Based Avatar Reconstruction: Optimization-based methods fit parametric head models or neural representations to multi-view captures or monocular videos, often using explicit 3D priors or tracked FLAME parameters.NeRF-based methods can produce high-quality, photorealistic head avatars but require hours to days of optimization per identity.
- Feed-Forward Avatar Reconstruction: Feed-forward methods predict avatars in a single forward pass, reducing the computational bottleneck of per-subject optimization.Early encoder-decoder approaches leverage 3DMM or FLAME priors for single-view reconstruction.
- Feed-Forward Avatar Reconstruction: Early feed-forward approaches lack photorealism or prioritize 3D face understanding over synthesis, while GPAvatar uses single-view input and precomputed FLAME parameters.The cited limitations motivate extending avatar models to multi-view inputs and removing external FLAME preprocessing.
3 Methodology
FFAvatar reconstructs animatable 3D Gaussian head avatars from few-shot unposed portraits by fusing multi-view information into a canonical representation and predicting FLAME parameters end-to-end. Its training combines a few-to-many objective, adversarial supervision, and a three-stage curriculum spanning scalable pretraining, multi-view fine-tuning, and optional personalization.
- Animation: The model anchors each Gaussian to a canonical FLAME vertex and animates only its center, leaving covariance, opacity, and color unchanged.Linear blend skinning supplies the deformation from expression, pose, and global head-pose parameters.
- Multi-view reconstruction: FFAvatar’s Multi-view Query-Former fuses geometry and texture cues from multiple unposed images into a consistent canonical Gaussian avatar.Canonical 3D queries attend to image tokens across views, producing identity-injected avatar features in canonical space.
- Animation: FFAvatar predicts per-view FLAME parameters directly from image embeddings, eliminating external FLAME preprocessing while retaining compatibility with explicit external tracking.The estimator predicts expression, pose, and global head-pose parameters, while FLAME skinning drives Gaussian centers during animation.
- Training objective: The few-to-many objective trains the canonical decoder from conditioning views to reconstruct multiple target views, aligning training with arbitrary-view avatar rendering.Each iteration samples disjoint conditioning and reconstruction subsets from an identity’s complete image set.
- Training losses: Adversarial supervision improves texture sharpness and overall rendering quality beyond pixel and perceptual losses alone.The projected discriminator uses differentiable augmentation to enhance texture fidelity and realism.
- Training curriculum: 60× faster than training from scratch, optional personalization typically converges in 500 optimization steps instead of around 100K.The three-stage curriculum progressively targets generalization, geometric fidelity, and identity preservation through scalable pretraining, multi-view fine-tuning, and personalization.
4 Experiments
Experiments show that FFAvatar generalizes strongly on NeRSemble, benefits substantially from its three-stage training and personalization, and enables fast end-to-end FLAME estimation. The model outperforms feed-forward baselines across reconstruction and identity metrics while preserving efficient deployment.
- Implementation and Training: FFAvatar is pretrained on MFHQ-1M with 1M identities, then fine-tuned on multi-view Ava256 captures before optional identity-specific personalization.The stages use 200K, 20K, and 500 optimization steps, respectively.
- Quantitative Comparison: FFAvatar outperforms feed-forward baselines across NeRSemble metrics, including +2.57 PSNR over GAGAvatar and +0.21 CSIM in the single-view setting.Evaluation uses 45 identities and 16 camera views, reporting PSNR, SSIM, LPIPS, and ArcFace CSIM.
- Qualitative Comparison: Qualitatively, FFAvatar improves realism and fidelity over baselines, while four input views provide richer appearance and geometry cues than single-view reconstruction.GAGAvatar produces overly smoothed geometry and pose misalignment, whereas LAM struggles with non-frontal and extreme poses.
- Personalization Analysis: 500-step personalization yields the best NeRSemble reconstruction results by rapidly adapting the feed-forward representation to identity-specific details.Feed-forward initialization converges within 500 steps, whereas random initialization remains blurry and preserves identity poorly.
- Ablation Study: -8.36 PSNR and -0.35 CSIM result from removing scalable pretraining, the largest ablation drop and evidence of its importance for unseen-identity generalization.Removing multi-view fine-tuning causes -3.53 PSNR and -0.13 CSIM, while removing GAN or few-to-many losses worsens LPIPS and CSIM.
- FLAME Estimator Analysis: 60 FPS versus 0.3 FPS makes the learned FLAME estimator over 200× faster than offline VHAP tracking while maintaining on-par personalized rendering quality.The estimator also supports explicit FLAME coefficient driving through the avatar’s LBS interface.
5 Limitation Analysis
FFAvatar remains limited by FLAME’s animation space, sparse-view ambiguity, and the loss of fine-grained identity details without personalization. These limitations can cause missing facial geometry, hallucinated boundaries, and artifacts in extreme novel views.
- Animation prior: FLAME’s solution space limits detailed modeling of eye gaze, mouth interiors, and tongue geometry.The animation prior cannot represent these facial details fully.
- Sparse-view ambiguity: Sparse input views may miss hair, neck, and clothing boundaries, requiring hallucination that can introduce extreme-view artifacts.The ambiguity is especially problematic under novel viewpoints far from the observed views.
- Identity fidelity: Without personalization, single-step sparse-view prediction may smooth fine-grained identity details.Optional personalization is intended to address this fidelity limitation.
6 Conclusion
FFAvatar is a generalizable feed-forward framework for reconstructing animatable 3D Gaussian head avatars from few-shot portrait images. Its scalable training and lightweight personalization support identity generalization, subject-specific fidelity, geometric consistency, and real-time animation without precomputed FLAME parameters.
- Conclusion: FFAvatar reconstructs animatable 3D Gaussian head avatars directly from few-shot portrait images through scalable pretraining, multi-view fine-tuning, and optional lightweight personalization.This combination targets identity generalization, subject-specific fidelity, and geometric consistency across extreme viewpoints.
- Conclusion: 49 FPS animation is achieved on a single NVIDIA A100 GPU without precomputed FLAME parameters.The framework is positioned as a scalable foundation for controllable, real-time human avatar synthesis and more accessible digital human creation.
A Experiment Setup Details
The experiments evaluate FFAvatar’s generalization on unseen NeRSemble subjects using single-image and multi-view inputs, comparing it with feed-forward avatar-generation baselines under official single-view settings.
- Baselines: The comparison includes GAGAvatar and LAM, while Avat3r is excluded because its checkpoint and code are unavailable.GPAvatar and NeRF-based multi-view avatar methods are discussed but omitted from the main quantitative table.
- Benchmark & Metrics: 45 identities across 16 camera views form the NeRSemble test set for evaluating reconstruction of high-fidelity 3D head avatars from single-image and multi-view inputs.Rendering quality is assessed with three standard paired-image metrics.