Source-linked AI summary
Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, Daniel Cohen-Or
TL;DR
The paper addresses the limited structural control of unconstrained text-to-3D generation. It introduces Latent-NeRF with Sketch-Shape and Latent-Paint guidance, and reports effective constrained shape and texture generation through latent rendering. The authors also identify limitations involving directional prompt tweaking and stochastic variation across seeds.
Problem
Text-to-3D generation can produce impressive results but may lack the ability to guide or enforce a specific 3D structure.
Method
The paper adapts score distillation to latent diffusion by rendering NeRFs in latent space, adds soft Sketch-Shape occupancy guidance, and applies latent guidance directly to mesh texture maps.
Results
The evaluated guidance variants together with latent-based guidance offer a compelling solution for constrained 3D shape and texture generation.
Takeaways & Limitations
Latent-NeRF provides a framework for text- and shape-guided generation of 3D shapes and textures, including refinement of coarse geometry and colorization of given meshes.
Takeaways & Limitations
The approach can fail with directional prompt tweaking for certain objects, and result quality may vary significantly between random seeds.
Abstract
from arXiv · showhide
Text-guided image generation has progressed rapidly in recent years, inspiring major breakthroughs in text-guided shape generation. Recently, it has been shown that using score distillation, one can successfully text-guide a NeRF model to generate a 3D object. We adapt the score distillation to the publicly available, and computationally efficient, Latent Diffusion Models, which apply the entire diffusion process in a compact latent space of a pretrained autoencoder. As NeRFs operate in image space, a naive solution for guiding them with latent score distillation would require encoding to the latent space at each guidance step. Instead, we propose to bring the NeRF to the latent space, resulting in a Latent-NeRF. Analyzing our Latent-NeRF, we show that while Text-to-3D models can generate impressive results, they are inherently unconstrained and may lack the ability to guide or enforce a specific 3D structure. To assist and direct the 3D generation, we propose to guide our Latent-NeRF using a Sketch-Shape: an abstract geometry that defines the coarse structure of the desired object. Then, we present means to integrate such a constraint directly into a Latent-NeRF. This unique combination of text and shape guidance allows for increased control over the generation process. We also show that latent score distillation can be successfully applied directly on 3D meshes. This allows for generating high-quality textures on a given geometry. Our experiments validate the power of our different forms of guidance and the efficiency of using latent rendering. Implementation is available at https://github.com/eladrich/latent-nerf
1. Introduction
The paper adapts score distillation to latent diffusion by optimizing a NeRF directly in latent space, then adds shape guidance for more controlled 3D generation and mesh texturing. Sketch-Shape constrains coarse geometry, while Latent-Paint applies guidance directly to a given mesh.
- Latent-NeRF: Latent-NeRF operates directly in a latent diffusion model’s compact latent space, avoiding repeated encoding of rendered RGB images during guidance.A learned linear layer can later transform the trained Latent-NeRF back into a regular NeRF for RGB refinement and additional constraints.
- Sketch-Shape Guidance: Sketch-Shape guidance uses a coarse 3D model to softly constrain NeRF occupancy while allowing text-guided refinement of geometry and texture.The constraint can be tuned for different strictness levels, letting users define base geometry while the model adds details.
- Latent-Paint: Latent-Paint applies latent score-distillation guidance directly to a given 3D mesh by optimizing its texture map through rendered views.Because the mesh structure and parameterization are fixed, the method targets texture generation on known geometry.
- Evaluation: The paper evaluates its guidance variants across multiple scenarios and reports a compelling solution for constrained 3D shape and texture generation.The reported framework combines latent-based guidance with different forms of shape control.
2. Related Work
Prior work studies 3D shape generation with direct 3D supervision, image-based supervision, and text-guided neural scene representations. The paper is positioned among approaches using differentiable rendering and diffusion or language-image models, without requiring a dataset.
- 3D Shape Generation: 3D shape synthesis has used implicit functions, meshes, point clouds, and image-based supervision, partly because 3D supervision is difficult to obtain.Some methods also choose 2D renderings as the supervising primitive even when 3D data is available.
- Text-to-3D with 2D Supervision: CLIP-Forge combines a shape-conditioned implicit autoencoder with a normalizing flow over shape codes to infer shapes from text through CLIP embeddings.Its text-to-shape capability relies on CLIP’s joint text-image embedding space.
- Neural Rendering: Neural rendering work uses differentiable NeRFs for novel-view generation, relighting, reconstruction, and additional neural objectives for 3D modeling.Differentiability permits objectives beyond conventional rendering during training.
3. Method
Latent-NeRF brings score distillation into Stable Diffusion’s latent space by optimizing a NeRF that renders latent feature maps. The method adds RGB refinement and shape-guidance controls, including coarse Sketch-Shape constraints and direct texture optimization on meshes.
- Latent diffusion and score distillation: Latent diffusion models denoise compact latent codes from a pretrained autoencoder rather than high-resolution images directly.The autoencoder uses reconstruction, perceptual, and patch-based adversarial losses during training.
- Latent diffusion and score distillation: Score distillation noises a rendered image, predicts the added noise with a diffusion model, and propagates the resulting pixel gradients to NeRF parameters.The denoiser is conditioned on an optional text prompt, and the gradients are approximated without back-propagating through the diffusion process.
- Latent-NeRF: Latent-NeRF optimizes a NeRF to render four-channel feature maps in Stable Diffusion’s latent space, alongside a volume density field.This representation uses NeRF rendering to impose spatial consistency across views.
- Latent-NeRF: Latent-NeRF’s score-distillation gradients are approximated through a single denoiser forward pass, while a sparsity loss penalizes ill-defined background masks and floating radiance clouds.The sparsity term encourages strict blending between the object NeRF and background NeRF.
- RGB refinement: A latent-trained NeRF can be converted to RGB by adding a linear layer initialized from an approximate latent-to-RGB mapping, then jointly fine-tuning the model.The RGB conversion enables refinement with pixel-space supervision and supports additional RGB-based guidance or shading constraints.
- Shape guidance: Sketch-Shape guidance softly matches NeRF occupancy to a coarse primitive-based geometry while allowing score distillation to add details near the surface.The leniency is controlled by σS: lower values impose a tighter constraint, and the loss is evaluated on points already sampled for rendering.
- Shape guidance: Latent-Paint applies score distillation directly to a mesh by optimizing a four-channel latent texture map through a differentiable renderer.The mesh is rendered from random viewpoints into 64 × 64 × 4 feature maps, and the latent texture is decoded to obtain the final RGB texture.
4. Evaluation
The evaluation demonstrates Latent-NeRF’s efficiency and qualitative performance, while shape-guidance improves control over generated geometry and Latent-Paint produces textures on varied meshes.
- Efficiency: Less than 15 minutes on a single V100, Latent-NeRF converges faster than RGB-NeRF at about 30 minutes and DreamFusion at about 1.5 hours on 4 TPUs.The comparison attributes RGB-NeRF’s overhead to encoding into latent space during guidance.
- Text-Guided Generation: Latent-NeRF produces consistent learned shapes across viewpoints and qualitatively outperforms DreamFields and CLIPMesh in the reported comparison.The authors cannot validate DreamFusion’s quality comparison because Imagen is not publicly available.
- RGB Refinement: RGB refinement improves quality mainly for complex objects or detailed textures, but makes iterations about ×2 slower and raises runtime above 30 minutes.The refinement can be applied after latent-space shape convergence as an optional polishing step.
- Text-Guided Generation: Textual Inversion enables Latent-NeRF to condition object generation on specific objects and styles defined only by input images.The conditioning is implemented through tokens learned from the input images.
- Sketch-Shape Guidance: Different text prompts steer the same Sketch-Shape toward refined geometries matching the prompt, and shape-guidance improves results compared with using the same prompts without it.The method works with varied Sketch-Shapes, including shapes extruded from 2D sketches.
- Latent-Paint: Latent-Paint generates textures for meshes with automatically computed or preexisting UV parameterizations and achieves more precise textures than Tango and CLIPMesh.UV maps improve texture granularity independence from geometric resolution and compatibility with MeshLab and Blender.
5. Limitations
The framework has several limitations: directional prompt assistance can fail for some objects, Stable Diffusion may produce poor directional images, and results vary across random seeds.
- Directional prompt tweaking can fail for certain objects when used to obtain plausible 3D shapes.The method adds camera-relative directions such as “front” or “side” to the text prompt.
- Stable Diffusion may generate unsatisfactory images when the desired viewing direction is specified.
- A generated squirrel can contain two faces from some viewing directions.The reported example attributes this issue to Stable Diffusion failing to generate squirrel back views.
- As σS grows, the Sketch-Shape constraint becomes more lenient and permits the shape to evolve into new geometries.
- Output quality can vary significantly between different random seeds because diffusion models behave stochastically.
6. Conclusions
The paper introduces a latent framework for text- and shape-guided 3D shape and texture generation. Its Latent-NeRF renders directly in latent space, while Sketch-Shape and Latent-Paint provide additional control over geometry and textures.
- The framework adapts score distillation to latent diffusion models for fast and flexible 3D object generation.
- Sketch-Shape and Latent-Paint provide two forms of shape-guided generation for additional control over the process.
- Latent-NeRF renders directly into a neural model’s latent space rather than pixel space.The paper argues that compact latent representations may support further latent-space rendering methods.