Source-linked AI summary
Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation
Rui Chen, Yongwei Chen, Ningxin Jiao, Kui Jia
TL;DR
Existing text-to-3D methods couple geometry and appearance, limiting fine-surface recovery and photorealistic rendering. Fantasia3D disentangles geometry and appearance using a hybrid representation, normal-map conditioning, and spatially varying BRDF materials; experiments report stronger high-quality and diverse content creation than existing methods.
Problem
Implicit text-to-3D representations couple geometry and appearance through volume rendering, limiting fine-surface recovery and photorealistic rendering for high-quality 3D assets.
Method
Fantasia3D disentangles geometry and appearance using DMTET-based hybrid geometry, normal-map conditioning for diffusion, and a spatially varying BRDF for material learning.
Results
Fantasia3D outperforms existing methods for high-quality and diverse 3D content creation across evaluated text-to-3D settings.
Takeaways & Limitations
The disentangled framework supports fine geometry, photorealistic rendering, and compatibility with relighting, editing, and physical simulation.
Takeaways & Limitations
Generating loose geometries such as hair, fur, and grass from text remains challenging, and the method primarily targets objects rather than complete scenes with backgrounds.
Abstract
from arXiv · showhide
Automatic 3D content creation has achieved rapid progress recently due to the availability of pre-trained, large language models and image diffusion models, forming the emerging topic of text-to-3D content creation. Existing text-to-3D methods commonly use implicit scene representations, which couple the geometry and appearance via volume rendering and are suboptimal in terms of recovering finer geometries and achieving photorealistic rendering; consequently, they are less effective for generating high-quality 3D assets. In this work, we propose a new method of Fantasia3D for high-quality text-to-3D content creation. Key to Fantasia3D is the disentangled modeling and learning of geometry and appearance. For geometry learning, we rely on a hybrid scene representation, and propose to encode surface normal extracted from the representation as the input of the image diffusion model. For appearance modeling, we introduce the spatially varying bidirectional reflectance distribution function (BRDF) into the text-to-3D task, and learn the surface material for photorealistic rendering of the generated surface. Our disentangled framework is more compatible with popular graphics engines, supporting relighting, editing, and physical simulation of the generated 3D assets. We conduct thorough experiments that show the advantages of our method over existing ones under different text-to-3D task settings. Project page and source codes: https://fantasia3d.github.io/.
1. Introduction
Fantasia3D targets high-quality text-to-3D generation by disentangling geometry from appearance, addressing limitations of implicit representations in fine surface recovery and photorealistic rendering. It combines hybrid geometry modeling, normal-map conditioning, and BRDF-based appearance learning, with reported advantages over existing methods.
- Motivation: Existing text-to-3D methods commonly use NeRF-based implicit representations that couple geometry and appearance, limiting fine surface, material, and texture recovery.Explicit and hybrid representations are proposed to improve surface modeling through explicit geometry and surface rendering.
- Method: Fantasia3D disentangles geometry and appearance learning to generate fine surfaces and rich materials or textures for photorealistic rendering.The framework is designed for high-quality text-to-3D content creation.
- Method: For geometry learning, Fantasia3D uses DMTET and conditions a pretrained image diffusion model on shape-encoded rendered normal maps rather than rendered color images.DMTET maintains a deformable tetrahedral grid with a differentiable mesh extraction layer, enabling explicit shape control.
- Capabilities: Fantasia3D supports user-controlled generation from text plus customized or generic shapes, and its disentangled assets support relighting, editing, and physical simulation.Experiments report advantages over existing methods for high-quality and diverse 3D content creation.
- Method: For appearance modeling, Fantasia3D introduces spatially varying BRDF learning into text-to-3D, supporting material learning and photorealistic rendering.The authors describe this as the first use of full BRDF learning for text-to-3D in their framework.
2. Related work
Prior text-to-3D methods generate geometry but provide limited exploration of high-quality lighting and surface materials, while material-focused methods may require an input mesh. Fantasia3D addresses these shortcomings by generating both surface geometry and corresponding materials from text.
- Text-to-3D content creation: Existing text-to-3D approaches use image diffusion, score distillation, novel-view synthesis, or text-conditioned 3D models, but explore high-quality lighting and surface materials only limitedly.These approaches include methods that generate 3D geometries or synthesize novel views.
- Surface material estimation: TANGO generates high-quality surface materials from text prompts but requires a 3D surface mesh as input.This requirement distinguishes it from methods intended to create both geometry and materials from text.
- Positioning: Fantasia3D generates high-quality surface geometries and corresponding materials, which the paper identifies as crucial for photorealistic rendering of generated 3D content.The authors describe this as closing the loop of object-level text-to-3D content creation.
3. Preliminary
Text-to-3D methods optimize differentiable 3D scene parameters using guidance from a pretrained diffusion model and Score Distillation Sampling. DMTET provides a hybrid representation with differentiable explicit-mesh extraction for iterative rendering.
- Score distillation sampling: DreamFusion optimizes differentiable scene parameters and synthesizes novel views from textual descriptions using a pretrained 2D diffusion model.The rendered image is produced by a differentiable generator parameterized by the scene variables.
- Score distillation sampling: Score Distillation Sampling uses the diffusion model’s predicted noise, the noisy rendered image, text embedding, and noise level to guide scene-parameter updates.The SDS gradient includes a weighting function and the image derivative with respect to scene parameters.
- Score distillation sampling: Fantasia3D uses Stable Diffusion as its released latent-space guidance model and revises the SDS loss accordingly.The paper notes that Imagen is not publicly accessible and defers implementation details to Section 4.2.
- DMTET: DMTET is a hybrid representation with a deformable tetrahedral grid and a differentiable Marching Tetrahedral layer for extracting explicit meshes during training.The representation predicts signed-distance values and vertex position offsets, then renders the extracted mesh with a differentiable renderer.
4. The Proposed Method
Fantasia3D separates geometry generation from appearance generation using DMTET, rendered normal maps, and a spatially varying PBR material model. The resulting pipeline connects geometry and appearance to diffusion guidance while supporting textured, photorealistic surface rendering.
- Geometry modeling: DMTET predicts signed distance values and vertex offsets, from which a differentiably extracted triangular mesh represents the generated geometry.The representation can be initialized from either a 3D ellipsoid or a user-provided customized model.
- Geometry modeling: Rendered normal maps, together with an early-training object mask, provide the shape-encoding input used to guide geometry with Stable Diffusion.Camera poses are sampled uniformly on the sphere, and SDS gradients update the geometry parameters through the differentiable renderer.
- Appearance modeling: The material model predicts diffuse kd, roughness-metallic krm, and tangent-space normal variation kn as spatially varying surface properties.The material MLP maps positional encodings of surface points to these parameters, which determine the BRDF used for rendering.
- Appearance modeling: Surface appearance is rendered with a BRDF-based illumination equation that combines diffuse and specular intensities under an environment map.The rendered image is formed by aggregating pixel colors along camera directions and is connected to the image encoder for SDS-based material learning.
- Texturing: UV edge padding expands UV-island boundaries to remove background pixels and white texture seams before textures are used in graphics engines.The method shifts SDS weighting during appearance training to mitigate over-saturated color in the later rendering pipeline.
5. Experiments
Fantasia3D is evaluated through ablations, zero-shot and user-guided generation, and compatibility with graphics-engine workflows. The experiments examine disentangled geometry and appearance, surface quality, texturing, and downstream editing, simulation, and relighting.
- Experimental settings: The experiments assess disentangled geometry and appearance through ablations, zero-shot generation, user-guided generation, and graphics-engine compatibility.The evaluation includes comparisons with DreamFusion, Magic3D, and Latent-NeRF, plus scene editing and simulation demonstrations.
- Ablation studies: Replacing normal-map inputs with fixed-material shaded images produces twisted geometries, while staged low- and high-resolution normal rendering balances convergence and fine-detail recovery.The method uses 64 × 64 × 4 normal-mask images early and 512 × 512 × 3 normal images later.
- Ablation studies: Replacing the full BRDF with diffuse rendering makes appearances less realistic and removes reflection effects across views.This ablation highlights the role of spatially varying material modeling in appearance generation.
- Zero-shot generation: In zero-shot generation, Fantasia3D produces more photorealistic results than Magic3D with competitive geometries, consistently outperforming DreamFusion in appearance and geometry generation.Its appearance stage also outperforms Text2Mesh, CLIP-Mesh, and Latent-NeRF in realistic appearance generation.
- User-guided generation: With customized meshes as initialization, Fantasia3D generates rich geometric and appearance details and outperforms Latent-NeRF under matched meshes and text prompts.The user-guided experiments use low-quality meshes from Text2Mesh, Latent-NeRF, and Stable DreamFusion.
- Scene editing and simulation: Generated assets support material editing, physical simulation, scene interaction, natural shadows, and relighting in Blender.Demonstrations include cloth and soft-body simulation, imported objects in the Cornell Box, and varied HDR environment lighting on Iron Man.
6. Limitations
Fantasia3D remains limited in generating loose geometries and complete scenes from text prompts. The authors identify hair, fur, grass, and backgrounds as important future targets.
- Scope limitations: Generating loose geometries such as hair, fur, and grass solely from text prompts remains a significant challenge.The method can produce loose visual effects but not corresponding loose geometries reliably.
- Scope limitations: Fantasia3D primarily emphasizes object generation and lacks the capacity to generate complete scenes with backgrounds from text prompts.Future work targets complete-scene generation alongside more intricate loose geometries.
7. Conclusion
The conclusion presents Fantasia3D as a text-to-3D method that disentangles geometry and appearance using DMTET and diffusion-based normal-map shape encoding. It supports rich materials, user-controlled initialization, relighting, editing, and physical simulation, while future work considers direct 3D diffusion from language models.
- Contributions: Fantasia3D disentangles geometry and appearance to generate fine surfaces and rich materials or textures for automatic text-to-3D creation.The method is based on the hybrid DMTET scene representation.
- Method: Geometry learning uses rendered normal maps as shape-encoded inputs to a pre-trained Stable Diffusion model, while appearance learning introduces spatially varying BRDFs.The BRDF enables material learning for photorealistic rendering of the learned surface.
- Applications: The generated assets support relighting, editing, and physical simulation in compatible graphics-engine workflows.These capabilities follow from the disentangled generation of geometry and appearance.
- Future work: Future research will investigate learning 3D diffusion directly from large language models.The current method relies on pre-trained image diffusion models, specifically Stable Diffusion.