Source-linked AI summary
InstantAvatar: Learning Avatars from Monocular Video in 60 Seconds
Tianjian Jiang, Xu Chen, Jie Song, Otmar Hilliges
TL;DR
Personalized high-fidelity avatars traditionally require calibrated multi-camera systems and substantial computation, motivating fast monocular reconstruction. InstantAvatar combines efficient neural representation, articulation, and dynamic-scene rendering techniques to reconstruct animatable avatars within 60 seconds; it achieves on-par image quality while training and inference are significantly faster than SoTA methods.
Problem
Personalized high-fidelity avatar acquisition typically requires calibrated multi-camera systems and significant computation, motivating lightweight, fast reconstruction from monocular video.
Method
InstantAvatar combines Instant-NGP, Fast-SNARF, dynamic-scene empty space skipping, and occupancy-aware regularization for efficient avatar reconstruction and rendering.
Results
Within 60 seconds, InstantAvatar reconstructs animatable avatars with on-par image quality while training and inference are significantly faster than SoTA methods.
Takeaways & Limitations
The system supports interactive avatar animation and rendering and substantially improves performance under equal time budgets compared with SoTA methods.
Takeaways & Limitations
Because reconstruction relies purely on image observations, the method cannot infer unseen regions, such as a subject’s back when only the front is captured.
Abstract
from arXiv · showhide
In this paper, we take a significant step towards real-world applicability of monocular neural avatar reconstruction by contributing InstantAvatar, a system that can reconstruct human avatars from a monocular video within seconds, and these avatars can be animated and rendered at an interactive rate. To achieve this efficiency we propose a carefully designed and engineered system, that leverages emerging acceleration structures for neural fields, in combination with an efficient empty space-skipping strategy for dynamic scenes. We also contribute an efficient implementation that we will make available for research purposes. Compared to existing methods, InstantAvatar converges 130x faster and can be trained in minutes instead of hours. It achieves comparable or even better reconstruction quality and novel pose synthesis results. When given the same time budget, our method significantly outperforms SoTA methods. InstantAvatar can yield acceptable visual quality in as little as 10 seconds training time.
1 ETH Z¨urich 2 Max Planck Institute for Intelligent Systems, T¨ubingen https://tijiang13.github.io/InstantAvatar/
InstantAvatar reconstructs animatable, high-fidelity human avatars from monocular video within 60 seconds and supports interactive animation and rendering.
- 60 seconds: InstantAvatar reconstructs animatable high-fidelity human avatars from monocular video.The system provides poses and masks for the reconstruction.
- 15 FPS at 540 × 540 resolution: the learned avatar can be animated and rendered interactively.
- Accelerated neural radiance fields, fast articulation correspondence search, and dynamic-scene empty-space skipping support near-instant avatar learning.
1. Introduction
InstantAvatar targets monocular avatar reconstruction that is fast enough for practical, walk-up use while retaining high-fidelity quality and interactive rendering. It combines efficient neural representation, articulation, and dynamic-scene rendering strategies to achieve this goal.
- Motivation: High-fidelity personalized avatars matter for tele-presence, AR/VR, 3D graphics, and the metaverse, but conventional acquisition often requires calibrated multi-camera systems and substantial computation.
- Motivation: Existing monocular neural avatar methods require hours of training and cannot render at interactive rates because they combine differentiable deformation with volume rendering.
- InstantAvatar: 60 seconds: InstantAvatar reconstructs high-fidelity avatars from monocular video, pose parameters, and masks, then animates and renders them at interactive rates.
- InstantAvatar: Instant-NGP provides an efficient neural representation, while Fast-SNARF enables learning from posed observations and avatar animation.
- Efficiency design: Dynamic human rendering becomes the bottleneck after acceleration structures and articulation modules are introduced, because standard volume rendering queries hundreds of samples per ray.
- Efficiency design: An articulation-aware occupancy grid skips empty space during rendering, using pose-specific grids at inference and a union grid across training frames.
- Evaluation: Minutes instead of more than 10 hours: the method achieves on-par reconstruction quality and better animation quality than state-of-the-art methods.
2. Related Work
Prior 3D human reconstruction methods rely on dense sensing or template meshes, while neural representations enable high-fidelity monocular avatars. InstantAvatar extends efficient rigid-scene representations to dynamic articulated humans.
- 3D Human Reconstruction: Dense camera arrays or depth sensors can achieve high-quality 3D human reconstruction, but their hardware requirements limit use to professional settings.
- 3D Human Reconstruction: Monocular methods commonly deform personalized or generic SMPL templates to fit 2D joints and silhouettes.
- 3D Human Reconstruction: Personalized templates may be unavailable, while generic templates cannot represent high-fidelity details and varied clothing topologies.
- Neural Representations: Neural representations reconstruct high-fidelity human avatars from sparse views or monocular video in a pose-independent canonical space.
- Accelerating Neural Radiance Fields: Voxel grids and multi-resolution hash tables replace MLPs to accelerate neural-field training and inference, with Instant-NGP improving memory efficiency and high-frequency detail storage.
- InstantAvatar: Instant-NGP and related acceleration methods target rigid objects, so InstantAvatar combines Instant-NGP with articulation and an empty-space skinning scheme for dynamic articulated humans.
3. Method
InstantAvatar combines accelerated canonical radiance fields, efficient articulation, and dynamic-scene empty-space skipping to reconstruct and render animatable human avatars efficiently.
- Canonical Radiance Field: InstantAvatar models human shape and appearance in canonical space with an Instant-NGP hash-grid radiance field decoded by a shallow MLP.The hash table stores multiscale feature grids that are interpolated and concatenated before decoding.
- Articulating Radiance Fields: A low-resolution skinning-weight field and bone transformations deform canonical points into posed space through linear blend skinning.The articulation module supports learning from posed images and animation of the learned avatar.
- Rendering Radiance Fields: The renderer maps posed samples back to canonical space, queries the canonical radiance field, and accumulates color and density along each ray.This posed-to-canonical mapping enables volume rendering of the articulated radiance field.
- Empty Space Skipping for Dynamic Objects: A normalized-space occupancy grid removes global orientation and translation before filtering empty-space samples for dynamic human scenes.At inference, a 64 × 64 × 64 posed-space grid is thresholded and dilated, and samples in non-occupied cells receive zero density without radiance-field queries.
- Empty Space Skipping for Dynamic Objects: During training, a shared occupancy grid records the union of occupied regions across frames and is updated periodically using moving averages.Defining the grid in normalized space makes the union tighter and further reduces unnecessary queries.
- Training Losses: Training minimizes robust pixel-color and mask losses, hard-surface regularization, and occupancy-based regularization that favors zero density in empty cells.Solid-surface regularization also permits early ray termination once accumulated opacity reaches 1, while occupancy regularization avoids stronger SMPL shape assumptions for loose clothing.
4. Experiments
Experiments evaluate InstantAvatar on real and synthetic monocular-human videos, measuring reconstruction quality, speed, novel-pose synthesis, and component effects against established baselines.
- Experimental Setup: PeopleSnapshot experiments follow Anim-NeRF’s protocol, using optimized pose parameters that remain frozen during InstantAvatar training for fair comparison.
- Experimental Setup: SURREAL provides challenging out-of-distribution poses by driving textured SMPL sequences with PeopleSnapshot training poses and synthetic test poses.
- Reconstruction Quality: At convergence, InstantAvatar produces significantly better generated images than Neural Body and on-par quality with Anim-NeRF on PeopleSnapshot.
- Speed: 1 minute: InstantAvatar trains on one RTX 3090, versus 13 hours on 2× RTX 3090 for Anim-NeRF and 14 hours on 4× RTX 2080 for Neural Body.
- Speed: 15 FPS: InstantAvatar renders 540 × 540 images on one RTX 3090, with meaningful appearance at 5s and acceptable quality at 10s.
- Novel Pose Synthesis Quality: InstantAvatar faithfully synthesizes challenging novel poses, while Anim-NeRF shows artifacts near adjacent body parts and performs worse for loose clothing.
- Ablation Study: Empty-space skipping improves training and rendering speed, while occupancy-based regularization reduces floating artifacts and improves PSNR without the degraded quality of global sparsity.
5. Conclusion
InstantAvatar reconstructs animatable human avatars from monocular video within 60 seconds and renders them at interactive rates. It combines efficient neural representations and articulation with dynamic-scene acceleration, while remaining unable to infer unseen regions.
- InstantAvatar reconstructs animatable human avatars from monocular videos within 60 seconds and renders them afterward at 15 FPS.The system combines Instant-NGP with Fast-SNARF, plus additional efficiency mechanisms.
- The method combines Instant-NGP and Fast-SNARF with empty space skipping and occupancy-aware regularization.These components improve rendering speed and reduce floating artifacts in space.
- Compared with state-of-the-art methods, InstantAvatar achieves on-par image quality while training and inference are significantly faster.The method requires minutes of training instead of more than 10 hours and outperforms SoTA methods under the same time budget.
- The method cannot infer unseen regions because reconstruction relies purely on image observations.A video capturing only the subject’s front side cannot reconstruct the back side.
- Future work could use learning-based methods to predict unobserved texture and geometry and extend the approach to articulated objects or animals.