Source-linked AI summary
LHM: Large Animatable Human Reconstruction Model from a Single Image in Seconds
Lingteng Qiu, Xiaodong Gu, Peihao Li, Qi Zuo, Weichao Shen, Junfei Zhang, Kejie Qiu, Weihao Yuan, Guanying Chen, Zilong Dong, Liefeng Bo
TL;DR
Single-image animatable human reconstruction must resolve coupled geometry, appearance, and deformation while overcoming limited high-quality rigged training data. LHM uses a feed-forward multimodal transformer with head feature pyramid encoding to predict Gaussian avatars, achieving strong reconstruction, generalization, and animation results in seconds. Its main scope boundary is limited training coverage of uncommon poses and extreme viewpoints in real-world video data.
Problem
Single-image animatable human reconstruction is challenged by coupled geometry, appearance, and deformation ambiguities and scarce high-quality rigged human training data.
Method
LHM uses a feed-forward multimodal transformer to predict canonical 3D Gaussian avatars, with head feature pyramid encoding for multi-scale facial features and video-based self-supervision.
Results
LHM achieves state-of-the-art reconstruction accuracy, generalization, and animation consistency on synthetic and in-the-wild datasets, producing animatable avatars in seconds.
Takeaways & Limitations
The model provides plausible animatable human avatars from single images without post-processing for face and hands.
Takeaways & Limitations
Real-world video training data has biased view distributions with limited uncommon poses and extreme-angle coverage, which can affect generalization to novel viewpoints.
Abstract
from arXiv · showhide
Animatable 3D human reconstruction from a single image is a challenging problem due to the ambiguity in decoupling geometry, appearance, and deformation. Recent advances in 3D human reconstruction mainly focus on static human modeling, and the reliance of using synthetic 3D scans for training limits their generalization ability. Conversely, optimization-based video methods achieve higher fidelity but demand controlled capture conditions and computationally intensive refinement processes. Motivated by the emergence of large reconstruction models for efficient static reconstruction, we propose LHM (Large Animatable Human Reconstruction Model) to infer high-fidelity avatars represented as 3D Gaussian splatting in a feed-forward pass. Our model leverages a multimodal transformer architecture to effectively encode the human body positional features and image features with attention mechanism, enabling detailed preservation of clothing geometry and texture. To further boost the face identity preservation and fine detail recovery, we propose a head feature pyramid encoding scheme to aggregate multi-scale features of the head regions. Extensive experiments demonstrate that our LHM generates plausible animatable human in seconds without post-processing for face and hands, outperforming existing methods in both reconstruction accuracy and generalization ability.
1. Introduction
Single-image animatable human reconstruction must disentangle coupled geometry, appearance, and deformation while handling limited high-quality rigged training data. LHM addresses these challenges with a scalable feed-forward transformer trained from video data and reports strong generalization and animation consistency.
- Single-image animatable avatar creation remains difficult because geometry, appearance, and deformation are coupled ambiguities.
- Existing parametric models struggle with loose-clothing geometry and high-fidelity facial details, limiting avatar expressiveness.
- Extending large reconstruction models to animatable humans requires an architecture combining 3D representation with animation and a solution to scarce rigged training data.
- LHM predicts canonical animatable human avatars as 3D Gaussians from a single image in seconds using a scalable feed-forward transformer.The model uses SMPL-X surface points as geometric anchors for predicting the canonical representation.
- A multimodal transformer fuses 3D body-point and image features through attention, while head feature pyramid encoding aggregates multi-scale head features for facial detail recovery.Training transforms predicted canonical Gaussians into various poses and uses rendering losses and regularization with video data.
- Trained without rigged 3D data, LHM achieves state-of-the-art real-world performance in generalization and animation consistency.
2. Related Work
Single-image human reconstruction has progressed from rigid parametric models toward implicit, feed-forward, and multimodal approaches. For animatable avatars, LHM directly regresses 3D Gaussians without UV representations or face-and-hand post-processing.
- Single-Image Human Reconstruction: Parametric mesh methods provide body priors but often miss diverse clothing and fine surface details because of rigid representations.
- Single-Image Human Reconstruction: Large reconstruction models accelerate generalizable feed-forward object reconstruction, motivating transformer-based human-avatar methods.
- Animatable Human Generation: Video-based and text-to-3D methods improve consistency or enable avatar generation, but text-to-3D approaches rely on long optimization processes.
- Animatable Human Generation: Animatable human methods combine diffusion, transformer, Gaussian-splatting, and optimization stages to generate canonical or consistent avatars.
- Animatable Human Generation: Unlike prior approaches, LHM directly regresses 3D human Gaussians without UV representations or post-processing for face and hand refinement.
3. Preliminary
The preliminary representation combines SMPL-X body structure with 3D Gaussian Splatting, providing articulated human parameters and view-dependent Gaussian appearance.
- Human Parametric Model: SMPL and SMPL-X represent human body structure using deformation techniques derived from thousands of 3D body scans.
- Human Parametric Model: SMPL-X uses shape parameters β ∈R20 and pose parameters θ ∈R55×3 to define articulated body configurations and mesh deformation.
- 3D Gaussian Splatting: 3D Gaussian Splatting represents 3D information with anisotropic Gaussian primitives parameterized by position, scale, rotation, opacity, and appearance features.Spherical-harmonic appearance features account for view-dependent lighting during differentiable rendering.
4. Method
LHM directly predicts an animatable canonical 3D Gaussian avatar from one image, combining SMPL-X geometric anchors with body and head image features. Its multimodal transformer, head feature pyramid, pose-aware training, and canonical regularization target detailed reconstruction and stable animation.
- Architecture: LHM represents the avatar with 3D Gaussian Splatting and directly predicts a canonical animatable avatar from a single RGB image.The representation supports real-time photorealistic rendering and pose-controlled animation.
- Architecture: The MBHT integrates geometric tokens, body image tokens, and head image tokens through attention and part-aware interactions.Geometric tokens attend to image features, while body and head tokens receive region-specific processing.
- Training Objective: Training warps predicted canonical Gaussians to target views with LBS and optimizes photometric, mask, perceptual, and canonical-space regularization losses.Shape regularization discourages excessive Gaussian anisotropy, while positional anchoring keeps Gaussians near SMPL-X initialization within a 5.25cm threshold.
- Feature Encoding: SMPL-X surface points provide geometric tokens, while pretrained vision features encode body appearance and texture.The geometric points are positionally encoded and projected into transformer token space; body image tokens come from a frozen Sapiens-1B encoder.
- Feature Encoding: HFPE aggregates multi-scale head features because encoder downsampling can discard facial detail.Features from multiple DINOv2 transformer depths combine high-frequency texture with deeper geometric information.
- Transformer Training: Head-token shrinkage masks 0% to 50% of head tokens during training to improve body-context utilization while preserving head reconstruction fidelity.The masking regularizes the attention imbalance toward head-region features.
5. Experiments
The experiments describe large-scale in-the-wild training and comparisons against several single-image human reconstruction baselines. The supplied passages emphasize dataset construction, optimization scale, and baseline categories rather than quantitative results.
- Training Data: The in-the-wild training data comprise 301,733 single-person video sequences curated from 500K initial human-motion samples.Filtering removes multi-person interactions, occluded faces, and low-quality frames through manual inspection and automated thresholds.
- Implementation: Training uses AdamW with mixed precision, gradient clipping, weight decay, and distributed NVIDIA A100 computation for 40K iterations.The 500M, 700M, and 1B models require 78, 112, and 189 hours, respectively.
- Evaluation: The single-image reconstruction evaluation compares LHM with GTA, SIFU, and PSHuman among the described baseline methods.The supplied passage characterizes these baselines by recursive optimization, pixel-aligned feature extraction, and multi-view diffusion components.
Ours
LHM is evaluated against existing methods on synthetic static reconstruction, in-the-wild animation, and scalability and ablation settings. It achieves stronger animation metrics and benefits from larger models, expanded data, and the proposed transformer design.
- Qualitative and quantitative evaluation: LHM produces more accurate and photorealistic animation results than baseline methods on in-the-wild sequences.
- Qualitative and quantitative evaluation: Table 1 evaluates static 3D reconstruction on 200 synthetic datasets using PSNR, SSIM, LPIPS, and Face Consistency.Metrics for models trained on the same synthetic dataset are reported for fair comparison.
- Qualitative and quantitative evaluation: 3.322, 0.059, 0.063, and 0.018 are LHM’s gains over AniGS in PSNR, SSIM, LIPIS, and FC, respectively.The evaluation uses 200 in-the-wild video sequences and compares synthesized animations with corresponding ground-truth sequences.
- Scalability and ablations: Increasing model parameters correlates with improved performance, with LHM-1B reconstructing more accurately than LHM-0.5B, especially in face regions.
- Scalability and ablations: Performance improvement diminishes as dataset size increases.
- Scalability and ablations: The proposed transformer block improves over the vanilla MM-transformer by 1.576, 0.017, 0.056, and 0.011 in PSNR, SSIM, LIPIS, and FC.Shrinkage regularization further improves overall performance but slightly reduces face consistency.
6. Conclusion
LHM reconstructs animatable 3D human avatars from a single image in seconds using multimodal feature fusion and head-specific multi-scale encoding. Experiments support strong generalization, while biased real-world viewpoints limit novel-view robustness.
- Conclusion: A multimodal transformer fuses 3D positional and 2D image features through attention for joint geometric and visual reasoning.
- Limitations and future work: Biased view distributions in real-world videos limit coverage of uncommon poses and extreme angles, potentially affecting generalization to novel viewpoints.The authors propose improved training strategies and more diverse data as future work.
- Conclusion: HFPE aggregates multi-scale DINOv2 head features to address facial-detail loss caused by the head’s small image area and spatial downsampling.
- Conclusion: Canonical-space regularization reduces semi-transparent boundary artifacts and floating points around the reconstructed human.The spherical loss addresses distorted Gaussian shapes, while removing the close-as-possible loss produces noticeable floating points.
F. More Results
Additional results show reconstruction and animation across diverse appearances, clothing, and poses. The method produces high-fidelity animatable avatars in one forward pass with photorealistic rendering.
- More Results: Figures 10–11 show reconstruction and animation results for images containing diverse appearances, clothing, and poses.
- More Results: LHM enables high-fidelity animatable human avatar reconstruction in a single forward pass with photorealistic rendering.
- More Results: HFPE uses multi-scale facial feature extraction to support the reconstruction pipeline.
- More Results: Canonical-space shape regularization is evaluated through an ablation study.