Source-linked AI summary
Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion
Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, Baining Guo
TL;DR
Generating detailed 3D digital avatars is constrained by the memory and computational cost of 3D representations. Rodin addresses this with roll-out tri-plane diffusion, 3D-aware convolution, latent conditioning, and hierarchical synthesis, producing high-fidelity avatars with customization and text-guided editing. The authors identify sampling speed and the 3D data bottleneck as future challenges.
Problem
High-quality neural-volume avatars require rich details, but their 3D memory and computational costs are prohibitive.
Method
Rodin rolls tri-plane feature maps into a 2D plane for 3D-aware diffusion, using cross-plane convolution, latent conditioning, and hierarchical synthesis.
Results
Rodin generates high-quality 3D avatar renderings and high-fidelity geometry, including detailed hairstyles and glasses, with smooth latent interpolations.
Takeaways & Limitations
Rodin supports avatar generation from scratch, portrait or text customization, and text-guided semantic editing, lowering the barrier to personalized avatar creation.
Takeaways & Limitations
Future work should improve 3D diffusion sampling speed and leverage ample 2D data to mitigate the 3D data bottleneck.
Abstract
from arXiv · showhide
This paper presents a 3D generative model that uses diffusion models to automatically generate 3D digital avatars represented as neural radiance fields. A significant challenge in generating such avatars is that the memory and processing costs in 3D are prohibitive for producing the rich details required for high-quality avatars. To tackle this problem we propose the roll-out diffusion network (Rodin), which represents a neural radiance field as multiple 2D feature maps and rolls out these maps into a single 2D feature plane within which we perform 3D-aware diffusion. The Rodin model brings the much-needed computational efficiency while preserving the integrity of diffusion in 3D by using 3D-aware convolution that attends to projected features in the 2D feature plane according to their original relationship in 3D. We also use latent conditioning to orchestrate the feature generation for global coherence, leading to high-fidelity avatars and enabling their semantic editing based on text prompts. Finally, we use hierarchical synthesis to further enhance details. The 3D avatars generated by our model compare favorably with those produced by existing generative techniques. We can generate highly detailed avatars with realistic hairstyles and facial hair like beards. We also demonstrate 3D avatar generation from image or text as well as text-guided editability.
1. Introduction
Rodin targets automatic generation of detailed digital avatars as neural radiance fields despite prohibitive 3D memory and computation costs. It combines 3D-aware diffusion, latent conditioning, and hierarchical synthesis to support coherent, detailed, and editable avatars.
- Digital avatars are widely used, but their data is scarce because specialized 3D artists must painstakingly create them, especially hair and facial hair.
- Rodin represents avatars as neural radiance fields and rolls tri-plane feature maps into a single 2D plane for efficient 3D-aware diffusion.Tri-planes represent a volume using three axis-aligned orthogonal feature planes.
- 3D-aware convolution links corresponding projected features across tri-planes, synchronizing detail synthesis according to their shared 3D relationships.
- Latent conditioning orchestrates globally coherent feature generation, improving avatar quality and enabling semantic editing.An additional image encoder extracts the semantic latent vector used as the diffusion model’s conditional input.
- Hierarchical synthesis first generates 64 × 64 tri-planes and then diffusion-upsamples them to 256 × 256.The upsampler is trained with a patch-wise image-level loss.
- The model is trained on multi-view images of 100K diverse avatars and supports generation from scratch, portrait or text customization, and text-guided semantic editing.
2. Related Work
Prior 3D generative methods have struggled with complex, authentic, and high-quality assets. Rodin instead uses diffusion with 3D supervision and a hierarchical tri-plane pipeline.
- Diffusion models have shown strong generative ability in 2D, but their generative capability remains much less explored in 3D.
- Early GAN- and VAE-based methods model voxels, point clouds, meshes, or implicit representations but have not demonstrated complex 3D assets.
- 3D-aware GANs trained from 2D collections suffer from training instability and mode collapse, while large-angle authentic avatar generation remains challenging.
- Rodin’s overview uses latent conditioning, coarse-to-fine tri-plane diffusion, and 3D-aware operators to produce rendered multi-view outputs.The base model generates 64 × 64 tri-planes, which are upsampled to 256 × 256 before rendering and convolutional refinement.
- Pretrained 2D generative networks provide limited 3D knowledge and can produce blurry 3D results.
3. Approach
Rodin fits neural radiance fields as tri-plane features and learns their distribution with hierarchical diffusion. Its 3D-aware processing, latent conditioning, and multi-resolution fitting address cross-plane coherence, global consistency, and detail generation.
- 3.1. Robust 3D Representation Fitting: Tri-plane representation factorizes each neural volume into three orthogonal 2D feature planes, reducing memory while retaining explicit 3D information.A lightweight MLP decodes queried tri-plane features into density and view-dependent color for volumetric rendering.
- 3.1. Robust 3D Representation Fitting: Randomly scaling tri-planes during fitting produces multi-resolution features that a shared decoder can render robustly across continuous scales.This addresses the poor 64 × 64 rendering obtained when only 256 × 256 tri-planes are fitted.
- 3.2. Latent Conditioned 3D Diffusion Model: Rodin learns the distribution of tri-plane features with diffusion, first generating coarse 64 × 64 tri-planes and then using diffusion-based upsampling for higher-resolution detail.The diffusion process progressively denoises noisy tri-plane codes, with stochastic ancestral sampling used at inference.
- 3.2. Latent Conditioned 3D Diffusion Model: 3D-aware convolution preserves tri-plane geometry by attending to corresponding row and column features from the other orthogonal planes.Axis-wise pooling makes associated cross-plane features spatially aligned for 2D convolution, reducing artifacts and improving thin structures such as hair strands.
- 3.2. Latent Conditioned 3D Diffusion Model: Latent conditioning uses a shared semantic latent vector to orchestrate globally coherent feature generation and support semantic editing with text prompts.The latent embedding is also used for classifier-free guidance during conditional sampling.
4. Experiments
Experiments evaluate Rodin on synthetic multi-view avatars, quantitative and qualitative comparisons, component ablations, and applications including image- and text-guided generation and editing.
- Implementation Details: The dataset contains 100K synthetic 3D avatars, each rendered from 300 multi-view images with known camera poses.The avatars combine identities, expressions, hairstyles, and accessories for radiance-field reconstruction.
- Unconditional Generation Results: Rodin produces high-fidelity 3D renderings and geometry, including detailed glasses and hairstyles recovered into meshes from generated density fields.The mesh is extracted using marching cubes.
- Comparison: Rodin achieves significantly lower FID than the baseline methods in the quantitative comparison.FID is computed from CLIP features using 5K generated samples.
- Comparison: Visual comparisons report higher-quality multi-view renderings and geometry for glasses and hair than prior 3D-aware GANs, which show more artifacts.The paper attributes those artifacts to geometry ambiguity from simple image supervision.
- Ablation Study: Ablations find latent conditioning, roll-out tri-plane features, and 3D-aware convolution progressively improve synthesis quality, especially for thin structures such as hair and cloth texture.The study uses a plain 2D CNN with channel-wise concatenated tri-plane features as its baseline.
- Applications: Hierarchical synthesis improves results through diffusion upsampling and convolution refinement, while Rodin supports portrait inversion, text-to-avatar generation, and text-based semantic editing.Image conditioning preserves characteristics such as expression, hairstyle, and glasses; detailed text descriptions enable fine customization.
5. Conclusion
Rodin supports personalized 3D avatar creation from portraits or text while offering an efficient direction for coherent, 3D-aware diffusion. The paper identifies faster sampling and greater use of 2D data as future priorities.
- Rodin enables users to customize 3D avatars from a portrait or text, lowering the barrier to personalized avatar creation.
- The paper focuses on avatars, but states that Rodin’s main ideas may apply to diffusion models for general 3D scenes.
- Future work should improve 3D diffusion sampling speed and jointly leverage abundant 2D data to mitigate the 3D data bottleneck.
- Diffusion generates samples by reversing gradual Gaussian noising, with a denoising model iteratively reducing noise from xT toward x0.The forward process adds random noise across timesteps, while the learned reverse process predicts noise at each timestep.
B.1. Architectural Design and Training Details
The implementation combines U-Net-based diffusion with tri-plane roll-out, 3D-aware convolution, latent conditioning, and specialized fitting and editing procedures. Separate diffusion components support latent sampling and text-to-avatar generation.
- Architecture: The base model uses a 192-channel U-Net modified with tri-plane roll-out and 3D-aware convolution, while a CLIP ViT-B/32 encoder supplies conditioning.
- Architecture: The upsample model directly enlarges feature maps from 64 to 256 using one upsample layer, applying the proposed operations in each residual block.
- Training: Training uses AdamW at a learning rate of 5e-5, with batch sizes of 48 for the base model and 16 for the upsample model.
- Data and fitting: The avatar dataset is built from explicit 3D representations fitted to multi-view images, while synthetic rendering addresses the difficulty of collecting large, consistent face datasets.
- Data and fitting: Tri-plane fitting learns 256 × 256 × 32 × 3 spatial features per subject with a four-layer fully connected MLP decoder.
- Data and fitting: Fitting minimizes rendered-image mean squared error and adds sparsity and smoothness regularizers to improve reconstruction quality and reduce floating artifacts.
- Editing and generation: Text-guided editing adds a CLIP-derived direction to a conditioned latent, using source and target text prompts to generate edited avatars.
- Editing and generation: Additional diffusion models generate latent codes for unconditional diversity and image embeddings for text-to-avatar generation using 100K portrait-text pairs.
C. Additional Ablation Study and Analysis
Additional analyses examine tri-plane resolution, fitting-view count, intermediate denoising stages, and the contribution of the proposed architectural components.
- Tri-plane fitting: 256 × 256 tri-planes are empirically sufficient to represent a subject while limiting memory cost, although fitting quality increases with resolution.
- Architectural analysis: Tri-plane roll-out and 3D-aware convolution are both crucial for high-fidelity results, particularly for thin structures such as hair strands and clothing details.
- Tri-plane fitting: Fitting quality becomes almost saturated when using 300 different views for neural tri-plane reconstruction.
- Generation process: During reverse diffusion, coarse structures emerge first and progressively become more detailed across intermediate time steps.
C.3. Effect of 3D-aware Convolution
The ablation study finds that tri-plane roll-out and 3D-aware convolution improve high-quality tri-plane generation, especially for fine avatar structures.
- Tri-plane roll-out and 3D-aware convolution enable efficient 3D-aware diffusion through a 2D architecture.
- The full upsample model shows a clear improvement over the base configuration when both roll-out and 3D-aware convolution are included.
C.4. Nearest Neighbors Analysis
Rodin generates distinct avatars rather than simply memorizing training examples, and also supports single-image-conditioned avatar reconstruction evaluated on 1K test subjects.
- Nearest-neighbor analysis indicates that generated avatars do not simply memorize the training data.The analysis uses CLIP feature similarity to identify nearest training examples.
- Rodin reconstructs 3D avatars from a single reference image using the input image’s CLIP embedding as conditioning.Conditional generation is evaluated on 1K test subjects, each represented by 300 views.
- The conditional-generation evaluation compares reconstructed images with ground-truth synthetic images using reported metrics.The supplied passage identifies the evaluation setup but does not include the metric values.
D. Additional Visual Results
Additional results show that Rodin produces detailed and diverse 3D avatars, supports mesh extraction and portrait creation, and yields smooth latent interpolations.
- Random and uncurated samples show high-quality renderings, detailed geometry, and visually pleasing diversity.Meshes are extracted from generated density fields with marching cubes to assess geometry.
- Nearest-neighbor visualizations compare generated samples with training-data examples using CLIP feature similarity.
- Latent interpolation between generated avatars produces consistent results with smooth appearance transitions.The interpolation varies the latent condition z between two generated avatars.
- Rodin also creates 3D portraits from a single reference image.
E. Societal Impact
Rodin targets a lower-cost, customizable tool for creating digital avatars used across creative industries, while relying on synthetic data and acknowledging potential misuse risks.
- Rodin is intended to provide a low-cost, fast, and customizable creation experience for traditional digital avatars.The paper links this goal to applications in movies, games, the metaverse, and the broader 3D industry.
- Synthetic Blender-rendered training images avoid privacy and copyright concerns associated with collecting real photos.Synthetic data also gives control over variation and diversity and can reduce bias in existing face datasets.
- The authors caution that portrait reconstruction and text-based customization may be misused to spread disinformation.They suggest tags or watermarks when distributing generated photos as a possible mitigation.
- The model’s high-quality rendered outputs may require safeguards such as tags or watermarks during distribution.
- The work is presented as a promising design tool that could significantly reduce costs in traditional 3D modeling and rendering pipelines.