Source-linked AI summary
AvatarPointillist: AutoRegressive 4D Gaussian Avatarization
Hongyu Liu, Xuan Wang, Zijian Wu, Yating Wang, Ziyu Wan, Yue Ma, Runtao Liu, Boyao Zhou, Yujun Shen, Qifeng Chen
TL;DR
Single-image 4D avatar generation needs explicit geometry that preserves fine, identity-specific details without fixed-topology constraints. AvatarPointillist autoregressively generates Gaussian point clouds and then decodes them into rendering attributes, achieving stronger quantitative and qualitative results than prior methods.
Problem
Existing 3DGS avatar methods have limited fidelity for fine-grained and identity-specific details, partly because they use incomplete or fixed-topology geometry representations.
Method
AvatarPointillist uses a decoder-only Transformer to autoregressively generate Gaussian point clouds, then conditions a Gaussian decoder on generated geometry and AR latent features to predict rendering attributes.
Results
AvatarPointillist outperforms prior methods in quantitative metrics and visual quality across the reported avatar-generation evaluations.
Takeaways & Limitations
Autoregressive point-cloud generation removes fixed-topology constraints and supports dynamic Gaussian placement in complex, identity-specific regions.
Abstract
from arXiv · showhide
We introduce AvatarPointillist, a novel framework for generating dynamic 4D Gaussian avatars from a single portrait image. At the core of our method is a decoder-only Transformer that autoregressively generates a point cloud for 3D Gaussian Splatting. This sequential approach allows for precise, adaptive construction, dynamically adjusting point density and the total number of points based on the subject's complexity. During point generation, the AR model also jointly predicts per-point binding information, enabling realistic animation. After generation, a dedicated Gaussian decoder converts the points into complete, renderable Gaussian attributes. We demonstrate that conditioning the decoder on the latent features from the AR generator enables effective interaction between stages and markedly improves fidelity. Extensive experiments validate that AvatarPointillist produces high-quality, photorealistic, and controllable avatars. We believe this autoregressive formulation represents a new paradigm for avatar generation, and we will release our code inspire future research.
1. Introduction
AvatarPointillist addresses limitations in single-image 4D avatar generation by modeling explicit Gaussian geometry autoregressively rather than relying on fixed templates. Its two-stage design generates adaptive point clouds and decodes them into rendering attributes, improving reported avatar quality.
- Motivation: 4D avatar methods provide multi-view-consistent, animatable geometry, while 2D methods lack 3D structure and can struggle with extreme poses and geometric distortions.3DGS offers faster rendering with photorealistic results than NeRF-based representations.
- Motivation: Existing 3DGS avatar methods achieve good overall performance but have limited fidelity for fine-grained and identity-specific details.GAGAvatar lifts 2D features into 3D, while LAM relies on a fixed point-cloud template and constant point count.
- Approach: AvatarPointillist casts 3DGS avatar generation as an autoregressive task that learns point clouds from scratch instead of using fixed templates.Point-by-point generation adjusts Gaussian spatial distribution, density, and scale for geometrically complex regions.
- Approach: A separate Gaussian decoder converts generated point-cloud geometry into full rendering parameters, and conditioning it on AR latent features enhances rendering quality.The decoder predicts attributes such as color and opacity after sequential geometry generation.
- Results: AvatarPointillist significantly outperforms all baselines quantitatively and qualitatively, producing high-quality, photorealistic, and controllable avatars.The authors present autoregressive explicit avatar geometry as a promising research direction.
2. Related Works
Related work spans 2D animation, 3D-aware avatarization, fitting-based pipelines, and end-to-end generation. Autoregressive geometry methods motivate representing 3D structures as discrete sequences, while AvatarPointillist applies this paradigm to Gaussian avatar geometry.
- 3D-Aware Animatable Avatar: Fitting-based methods optimize meshes, NeRFs, SDFs, points, or 3D Gaussians per subject, often overfitting the input viewpoint and extrapolating poorly to novel views.Diffusion-assisted fitting improves identity generalization but remains time-consuming because optimization is still required.
- 3D-Aware Animatable Avatar: End-to-end methods learn priors from large-scale datasets to generate animatable avatars, with 3DGS providing faster rendering while retaining high visual quality.Existing 3DGS methods still face limitations such as auxiliary refinement networks or difficulty modeling unseen regions.
- Autoregressive Geometry Generation: Autoregressive geometry generation treats meshes, point clouds, or implicit representations as sequences of discrete tokens predicted by Transformers.VQ-VAE-based two-stage pipelines and hierarchical or coarse-to-fine strategies address discrete representation and long-sequence challenges.
- Autoregressive Geometry Generation: AvatarPointillist applies next-token prediction to Gaussian point-cloud generation and combines decoded positions with AR features to predict attributes and animate results.Binding information supports animation through Linear Blend Skinning.
3. Preliminary
The method constructs bound 3DGS training point clouds from fitted Gaussian avatars, orders them deterministically, and converts their continuous values into discrete sequences for autoregressive training. Each point combines spatial coordinates with a mesh-face binding index.
- Data Construction: Training data are built by fitting GaussianAvatars models to each Nersemble identity, producing Gaussians bound to specific FLAME mesh faces.Each Gaussian has local-space location, rotation, and scale that become dynamic globally as its parent triangle moves.
- Data Construction: Canonical Gaussian point clouds contain each point’s global coordinates and a binding index identifying the attached FLAME face.The point-cloud sequence is defined over N Gaussians.
- Quantization and Order of Coordinates: Points are sorted by y, then z, then x coordinates so identical point clouds produce identical model input sequences.This ordering is called a yzx sort order.
- Quantization and Order of Coordinates: Each point uses four sequence values—x, y, z, and binding index—with grouped start and end tokens preserving that structure.Four start-of-sequence and four end-of-sequence tokens are added around each sequence.
- Quantization and Order of Coordinates: Continuous coordinates are quantized into discrete bins, with 1024 levels selected as a balance between geometric precision and computational load.Coordinate tokens occupy values from 0 to 1023, while binding tokens are offset into a separate vocabulary region.
- Quantization and Order of Coordinates: The number of Gaussians varies across point clouds, with the reported range spanning 1024 to 11167 points.This variable length is compatible with the adaptive point-cloud representation.
4. Method
AvatarPointillist generates an animatable 4D Gaussian avatar from a single source image through autoregressive point-cloud generation followed by Gaussian attribute decoding. The method jointly models geometry, binding information, and rendering attributes in a two-stage pipeline.
- Overview: The method takes a source image and target motion, then generates a canonical 3D Gaussian Splatting point cloud for an animatable avatar.The AR model predicts the canonical point cloud, while the target individual's motion drives the resulting avatar.
- Autoregressive Model: A decoder-only Transformer autoregressively predicts each quantized token from all previously generated tokens.The sequence contains four tokens per point for x, y, z, and binding information.
- Autoregressive Model: The model injects DINOv2 image features and point-cloud features derived from FLAME parameters into the autoregressive generator.FLAME sample vertices are encoded as point-cloud features before concatenation with the image features.
- Gaussian Decoder: A Transformer-based Gaussian decoder combines point geometry with autoregressive hidden features to predict color, opacity, scale, rotation, and positional offsets.Predicted offsets are added to canonical positions for fine-grained geometric adjustment.
- Expression Animation: Binding information enables animation through barycentric interpolation of FLAME properties, followed by vertex-based LBS and corrective blendshapes.The rigged avatar is driven by FLAME pose and expression parameters.
- Training: Training first optimizes the autoregressive model, then freezes it while separately training the Gaussian decoder with rendering losses.The decoder uses L1, SSIM, LPIPS, and offset regularization, with λL1 = 1, λSSIM = 0.5, λLP IP S = 0.1 and λReg = 0.1.
5. Experiments
AvatarPointillist is evaluated against recent single-image 4D avatar methods using quantitative metrics, qualitative reenactment comparisons, and ablations of its autoregressive geometry and Gaussian-decoder inputs. It consistently outperforms baselines and benefits from generating adaptive point clouds and combining spatial coordinates with AR features.
- Experimental Setup: Experiments compare AvatarPointillist with two NeRF-based and two Gaussian Splatting-based single-image 4D avatar reconstruction methods.The baselines are AvatarArtist, Portrait4Dv2, LAM, and GAGAvatar.
- Experimental Setup: Evaluation measures perceptual quality with LPIPS and FID, expression accuracy with AKD, pose consistency with APD, and identity preservation with CLIPScore.
- Qualitative Results: Qualitative comparisons show more realistic reenactments with better pose and expression alignment, sharper details, and more identity-accurate outputs.Compared with baselines, the method better preserves hair texture and facial contours.
- Quantitative Results: AvatarPointillist consistently outperforms all baselines across metrics in both self- and cross-reenactment tasks, indicating stronger identity preservation and motion-transfer accuracy.The evaluation uses aligned input views and standardized source and target settings across methods.
- Ablation Study: The static FLAME-position baseline fails to capture subject-specific geometry, especially in hair, because it cannot adaptively allocate points to important regions.Its reliance on fixed points and limited per-point features also reduces identity consistency.
- Ablation Study: Combining positional encodings with AR features gives the Gaussian decoder spatial guidance and semantic cues, producing the best overall quality.Using only AR features lacks spatial information, while using only coordinates misses the semantic richness of AR features.
6. Conclusion
AvatarPointillist is a one-shot 4D Gaussian avatar framework that autoregressively generates Gaussian point clouds, removes fixed topology constraints, and uses a two-stage decoder to produce rendering attributes. Experiments report higher quantitative metrics and better visual quality than prior methods.
- AvatarPointillist generates Gaussian point clouds point by point, removing fixed topology constraints for one-shot 4D avatar generation.
- Dynamic control over Gaussian number and placement lets the model focus on complex, identity-specific regions while exploiting 3DGS adaptivity.
- The two-stage architecture feeds autoregressive outputs and hidden features into a Gaussian decoder that predicts high-quality rendering attributes.
- AvatarPointillist outperforms prior methods in both quantitative metrics and visual quality.